Cannot run upwelling test case

[mmagaldi]

Hello everybody,

I have recently donwloaded the trunk and installed Roms (Revision 1188) on a new system where all dependencies are compiled via Intel oneAPI DPC++/C++ Compiler version 2023.2.0 .

I would like first to run the upwelling test case in serial. I am able to compile Roms without problems (see summary below) obtaining the executable romsS file

Code: Select all

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
GNU Build script command:      build_roms.sh -j 4
ROMS source directory:         /home/marcello/RomsAug23
ROMS build  directory:         /home/marcello/RomsAug23/Projects/test_upwelling/expSer_Aug23/Build_roms
ROMS Application:              UPWELLING
Fortran compiler:              ifort
Fortran flags:                 -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

When trying to run the code, Roms does not go thru and I get the following weird error right at the beginning:

Code: Select all

--------------------------------------------------------------------------------
 Model Input Parameters:  ROMS/TOMS version 4.2  
                          Tuesday - August 8, 2023 -  3:04:00 PM
--------------------------------------------------------------------------------
forrtl: severe (153): allocatable array or pointer is not allocated
Image              PC                Routine            Line        Source             
romsS              0000000000804626  get_metadata_mod_         977  get_metadata.f90
romsS              00000000006D4CFA  mod_ncparam_mp_in        1165  mod_ncparam.f90
romsS              000000000060B49A  read_phypar_              240  read_phypar.f90
romsS              00000000005F3C69  inp_par_mod_mp_in          85  inp_par.f90
romsS              000000000040FD0F  roms_kernel_mod_m          86  roms_kernel.f90
romsS              000000000040F81B  MAIN__                     75  master.f90
romsS              000000000040F60D  Unknown               Unknown  Unknown
libc.so.6          00007F887CE29D90  Unknown               Unknown  Unknown
libc.so.6          00007F887CE29E40  __libc_start_main     Unknown  Unknown
romsS              000000000040F525  Unknown               Unknown  Unknown

I am making sure that the local roms_upwelling.in file points to the correct path for the varinfo.yaml file.

I am very likely doing something stupid and will appreciate any help.

Thanks in advance,
marcello

[arango]

Did you specified the correct location of varinfo.yaml? We recommend Users download the ROMS test repository. Everything is set up and isolated for you.

[mmagaldi]

Hi Hernan,

yes I am sure to have specified the correct location of varinfo.yaml.

I have also downloaded the test repo and used the build_roms.sh script under the test/upwelling directory together with header and input files in the same dir as indeed they are different from those under ROMS/External and ROMS/Include.

Nothing: exactly the same error.

I am clueless...

[mmagaldi]

It really beats me as the issue seems to arise after the

Code: Select all

CALL YML%destroy ()

at L985 of the get_metadata.F file having IsYaml defined as TRUE

[robertson]

I was able to reproduce your error by installing and using the same version of the Intel compiler (2023.2.0 released July 13, 2023). I tracked the issue in the TotalView debugger and appears to be a compiler bug. If you have access to an earlier version of the compiler I would try using that instead. Because of an unrelated issue (unexplained segmentation violation in a different piece of software) we have been using 2023.0.0. There is an update to the Fortran compiler (2023.2.1) that I will test and let you know if it fixes the problem.

[mmagaldi]

I was able to reproduce your error by installing and using the same version of the Intel compiler (2023.2.0 released July 13, 2023). I tracked the issue in the TotalView debugger and appears to be a compiler bug. If you have access to an earlier version of the compiler I would try using that instead. Because of an unrelated issue (unexplained segmentation violation in a different piece of software) we have been using 2023.0.0. There is an update to the Fortran compiler (2023.2.1) that I will test and let you know if it fixes the problem.

Thanks David. Good to know I wasn't doing something stupid. Will try the 2023.0.0 as soon as I have time to recompile all dependencies.
m.

[mmagaldi]

I was able to reproduce your error by installing and using the same version of the Intel compiler (2023.2.0 released July 13, 2023). I tracked the issue in the TotalView debugger and appears to be a compiler bug. If you have access to an earlier version of the compiler I would try using that instead. Because of an unrelated issue (unexplained segmentation violation in a different piece of software) we have been using 2023.0.0. There is an update to the Fortran compiler (2023.2.1) that I will test and let you know if it fixes the problem.

Ok different things:

• I did download both new patches for the C and Fortran compilers (2023.2.1) but unfortunately got the same error;

• I did download and use the (2023.0.0) version for the compilers to recompile all dependencies as suggested by David and everything is fine: the upwelling test case runs both in serial and parallel mode with openMPI;

• Not sure it is important but a very old version of ROMS that was already there in our systems (specifically Revision 854, July 2017) is also working fine and runs both in serial and parallel mode with the new (2023.2.0) Intel oneAPI DPC++/C++ Compiler version.

I will stick to the (2023.0.0) version. Thanks again for all your help!
Marcello

[arango]

Great. The versioning of the ifort is weird. I used 2023.0.0 in our Spack-Stack for ROMS-JEDI:

Code: Select all

pontus % which ifort
/opt/sw/apps/intel/oneapi/compiler/2023.0.0/linux/bin/intel64/ifort
pontus % ifort --version
ifort (IFORT) 2021.8.0 20221119
Copyright (C) 1985-2022 Intel Corporation.  All rights reserved.

We noticed that every time it is updated, they introduce compiler bugs. The YAML parser is a complicated modern code. I plan to add a self-contained program into ROMS/Programs to test the viability of compiler versions.

[foolishwhy]

Is there any solution for this issue? I've met the same problem. The upwelling case can be run with ROMS 3.9 (without yaml) on the same machine.

[robertson]

As of now, the only fix is to use 2023.0.0 or older. It is questionable whether Intel will fix this since they are phasing out the classic C/C++ (icc/icpc) and Fortran (ifort) compilers in favor of their newer LLVM based counterparts (icx, icpx, and ifx).

[foolishwhy]

Thanks. I will try older Intel compiler.

[robertson]

I am unsure what version actually corrected this compiler bug but I'm happy to report that Intel OneAPI 2024.0 (i.e. 2021.11.1 for the classic compilers) fixes this problem.

[pheidary]

Hello everyone,
I am working on a cray system, and I have these intel module to use:

Code: Select all

------------------------------------- /opt/cray/pe/lmod/modulefiles/mix_compilers --------------------------------------
   intel-classic-mixed/2022.1.0        intel-mixed/2024.0                 intel-oneapi-mixed/2024.0
   intel-classic-mixed/2023.2.0 (D)    intel-mixed/2024.2                 intel-oneapi-mixed/2024.2
   intel-mixed/2022.1.0                intel-oneapi-mixed/2022.1.0
   intel-mixed/2023.2.0         (D)    intel-oneapi-mixed/2023.2.0 (D)

------------------------------------------ /opt/cray/pe/lmod/modulefiles/core ------------------------------------------
   PrgEnv-intel/8.5.0            intel-oneapi/2022.1.0        intel-oneapi/2024.2        intel/2024.0
   intel-classic/2022.1.0        intel-oneapi/2023.2.0 (D)    intel/2022.1.0             intel/2024.2
   intel-classic/2023.2.0 (D)    intel-oneapi/2024.0          intel/2023.2.0      (D)

To compile ROMS with an intel compiler I load these modules:PrgEnv-intel/8.5.0, and also cray-netcdf, and cray-hdf. But when I compile and try to run wc13 test case, I get these errors:

Code: Select all

forrtl: severe (153): allocatable array or pointer is not allocated
Image              PC                Routine            Line        Source             
romsM              00000000008A709A  Unknown               Unknown  Unknown
romsM              0000000000502888  Unknown               Unknown  Unknown
romsM              0000000000730E25  Unknown               Unknown  Unknown
romsM              000000000042B8DC  Unknown               Unknown  Unknown
romsM              000000000040E981  Unknown               Unknown  Unknown
romsM              000000000040F043  Unknown               Unknown  Unknown
romsM              000000000040E23D  Unknown               Unknown  Unknown
libc-2.31.so       000014F7E0AFB24D  __libc_start_main     Unknown  Unknown
romsM              000000000040E16A  Unknown               Unknown  Unknown

I wonder if the intel version is really important and some versions are not compatible with ROMS.

I really appreciate any suggestions or help in this regard.

Best,
Parisa

[robertson]

After you have PrgEnv-intel/8.5.0, cray-netcdf, and cray-hdf loaded, what does module list return? Also, what is the output of the ifort --version command?

I would suggest you compile the code with USE_DEBUG=on in your build script so we can get a better sense of where in the code it is failing.

[pheidary]

After you have PrgEnv-intel/8.5.0, cray-netcdf, and cray-hdf loaded, what does module list return? Also, what is the output of the ifort --version command?

I would suggest you compile the code with USE_DEBUG=on in your build script so we can get a better sense of where in the code it is failing.

Hi, thank you for your reply. Here is my loaded module (module list):

Code: Select all

Currently Loaded Modules:
  1) craype-x86-genoa     4) perftools-base/23.12.0                          7) CmrsEnv/default      10) perlbrew/5.38.2  13) cray-dsmml/0.2.2     16) PrgEnv-intel/8.5.0
  2) libfabric/1.20.1     5) xpmem/2.9.6-1.1_20240510205610__g087dc11fc19d     8) TimeZoneEDT/default  11) intel/2023.2.0   14) cray-mpich/8.1.28    17) cray-hdf5/1.12.2.9
  3) craype-network-ofi   6) cray-pmi/6.1.13                                 9) DefApps/default      12) craype/2.7.30    15) cray-libsci/24.07.0  18) cray-netcdf/4.9.0.9

And the output of ifort --version is
ifort (IFORT) 2021.10.0 20230609
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

I also tred to debug using USE_DEBUG=on, and here is what I get:

forrtl: severe (153): allocatable array or pointer is not allocated
Image PC Routine Line Source
romsG 00000000020DE4F7 get_metadata_mod_ 977 get_metadata.f90
romsG 00000000009521E9 mod_ncparam_mp_in 1168 mod_ncparam.f90
romsG 0000000001AB11C2 read_phypar_ 243 read_phypar.f90
romsG 00000000004C053C inp_par_mod_mp_in 114 inp_par.f90
romsG 000000000040ECA0 roms_kernel_mod_m 131 roms_kernel.f90
romsG 0000000000411C60 MAIN__ 98 master.f90
romsG 000000000040EABD Unknown Unknown Unknown
libc-2.31.so 0000150D32A0924D __libc_start_main Unknown Unknown
romsG 000000000040E9EA Unknown Unknown Unknown

I keep investigating the issue, but if you have any other suggestions I would really appreciate it.

Thanks,
Parisa

[robertson]

Hi, thank you for your reply. Here is my loaded module (module list):

Code: Select all

Currently Loaded Modules:
  1) craype-x86-genoa     4) perftools-base/23.12.0                          7) CmrsEnv/default      10) perlbrew/5.38.2  13) cray-dsmml/0.2.2     16) PrgEnv-intel/8.5.0
  2) libfabric/1.20.1     5) xpmem/2.9.6-1.1_20240510205610__g087dc11fc19d     8) TimeZoneEDT/default  11) intel/2023.2.0   14) cray-mpich/8.1.28    17) cray-hdf5/1.12.2.9
  3) craype-network-ofi   6) cray-pmi/6.1.13                                 9) DefApps/default      12) craype/2.7.30    15) cray-libsci/24.07.0  18) cray-netcdf/4.9.0.9

And the output of ifort --version is
ifort (IFORT) 2021.10.0 20230609
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

Number 11 (intel/2023.2.0) in your module list is known to fail with modern versions of the ROMS source code. It is a compiler bug so there is really nothing we can do about it. As a first step, I would try the following commands from a fresh login:

Code: Select all

module load PrgEnv-intel/8.5.0 cray-hdf5 cray-netcdf
module swap intel/2024.0

With any luck, that will switch your compiler version only. Since it is not a major version change for the classic compilers ( 2021.10 to 2021.11) all compiled software should be compatible. However, if the sysadmins explicitly put dependencies in the module files this might not work. Give it a shot and let us know.

[pheidary]

Thanks, I loaded modules as you suggested:
Currently Loaded Modules:

Code: Select all

  1) craype-x86-genoa     4) perftools-base/23.12.0                          7) CmrsEnv/default      10) perlbrew/5.38.2   13) PrgEnv-intel/8.5.0  16) cray-libsci/24.07.0
  2) libfabric/1.20.1     5) xpmem/2.9.6-1.1_20240510205610__g087dc11fc19d      8)TimeZoneEDT/default  11) craype/2.7.30     14) intel/2024.0        17) cray-hdf5/1.14.3.1
  3) craype-network-ofi   6) cray-pmi/6.1.13                                 9) DefApps/default      12) cray-dsmml/0.2.2  15) cray-mpich/8.1.28   18) cray-netcdf/4.9.0.13

Now, I am not getting those errors, and this issue resolved. However this one also is like the other problem that I have (viewtopic.php?p=25729#p25729), freezes at the last inner iteration becasue it cannot calculate the eigenvalues and eigenvectors. Which I believe that is coming from the system and not ROMS.
Thank you so much, at least my question about intel version resolved.

Best,
Parisa