Missing openmpi directory during compiling

[JMette]

Hi all,

I have a question regarding the configuration of the upwelling test case with openmpi on our computing cluster. When I run in serial, everything works fine. Also, on my private computer, I'm able to run it with mpi in the default config. When I use openmpi, during compile time I get the following output:

Code: Select all

f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow -ffree-form -ffree-line-length-none mod_ice.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_stepping.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow -ffree-form -ffree-line-length-none round.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_eoscoef.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow lapack_mod.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow lubksb.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow ludcmp.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mp_routines.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow read_icepar.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow sqlq.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_param.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_iounits.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow nrutil.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_scalars.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow -ffree-form -ffree-line-length-none mod_sediment.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_behavior.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow shapiro.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow gasdev.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow -fno-strict-overflow ran_state.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow obs_k2z.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow ran1.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow exchange_2d.f90
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_parallel.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
mod_parallel.f90:19: Error: Can't open included file 'mpif.h'
make: *** [ROMS/Modules/Module.mk:15: /home/jmette/roms_project/roms_test/upwelling/Build_romsG/mod_parallel.o] Error 1
make: *** Waiting for unfinished jobs....

For what I understand from that error, the build script cannot find the openmpi installation, right? I wait right now for more information on that from the person responsible for the computation cluster. But then would be my question:

How do I specify the custom path in the build script? Is it this block and if yes, how would a working example look like (I'm not so experienced with bash usage & co)?

Code: Select all

#--------------------------------------------------------------------------
# If applicable, use my specified library paths.
#--------------------------------------------------------------------------

 export USE_MY_LIBS=no            # use system default library paths
#export USE_MY_LIBS=yes           # use my customized library paths

MY_PATHS=${COMPILERS}/my_build_paths.sh

if [ "${USE_MY_LIBS}" == "yes" ]; then
  source ${MY_PATHS} ${MY_PATHS}
fi

Attached is also my build script, I'm very glad for any help!

[wilkin]

ROMS should be able to find what it needs in the system defaults, so your sysadmin build of MPI might be problematic.

If you need to customize the paths, it's best to do it in ./Compilers/my_build_paths.csh which will be used when (in build_roms.sh) you activate ...

Code: Select all

export USE_MY_LIBS=yes           # use my customized library paths
MY_PATHS=${COMPILERS}/my_build_paths.sh

[JMette]

Thanks for the help. I also found now the corresponding wiki pages.
But the follow-up problem is, that the default paths for all libraries in the my_build_paths.sh are (understandably) not correct and it then cannot find e.g. the netcdf-library. If I understand it correctly, I have do define the paths for every library by hand in this script? Or is there a way to use only a custom path for the mpi installation and keep the other ones from the none-custom build script?

[JMette]

After some correspondence with John Wilkin & David Robertson I tried a few things and achieved some progress:

- I tried to

Code: Select all

export PATH=/path/to/mpi/bin:$PATH

but that also didn't worked out.

- I tried to compile a simple fortran program with mpf90 and that gave the same error message as above, so probably the openmpi-installation on the computing cluster has some flaws.

- With a new, additional openmpi-installation in the home directory of my user and after exporting the new "/openmpi/bin"-path, I was able to compile and run the fortran test program with mpi sucessfully as well as building the upwelling test case. For the test case I get the warning:

Code: Select all

distribute.f90:4179:26:

 4179 |       CALL mpi_allreduce (Asend, Areduce, Npts, MP_FLOAT, handle,       &
      |                          1
......
 4412 |         CALL mpi_allreduce (Asend, Areduce, Isize,                      &
      |                            2
Warning: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)
distribute.f90:4066:28:

 4066 |         CALL mpi_allreduce (Asend(i), Areduce(i), 1, MPI_INTEGER,       &
      |                            1
......
 4412 |         CALL mpi_allreduce (Asend, Areduce, Isize,                      &
      |                            2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
distribute.f90:3837:26:

 3837 |       CALL mpi_allgather (my_bounds, np, MPI_INTEGER,                   &
      |                          1
......
 5588 |       CALL mpi_allgather (Asend, Npts, MP_FLOAT, Arecv, Npts, MP_FLOAT, &
      |                          2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
distribute.f90:1408:22:

 1408 |       CALL mpi_bcast (ibuffer, Npts, MPI_INTEGER, MyMaster,             &
      |                      1
......
 5036 |       CALL mpi_bcast (A, Asize, MP_FLOAT, MyMaster, OCN_COMM_WORLD,     &
      |                      2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
distribute.f90:218:22:

  218 |       CALL mpi_bcast (A, Npts, MP_FLOAT, MyMaster, MyCOMM, MyError)
      |                      1
......
 5036 |       CALL mpi_bcast (A, Asize, MP_FLOAT, MyMaster, OCN_COMM_WORLD,     &
      |                      2
Warning: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)

- But when I run the upwelling test case, it gives the following error:

Code: Select all

[*** The MPI_Comm_f2c() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[kcn0:1779501] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
--------------------------------------------------------------------------
prterun detected that one or more processes exited with non-zero status,
thus causing the job to be terminated. The first process to do so was:

   Process name: [prterun-kcn0-1779497@1,1] Exit code:    14
--------------------------------------------------------------------------

As far as I found out, this error is caused during the linking process when components from different mpi versions get mixed up, so a function gets called before initialization.

- I then also exported

Code: Select all

export LD_LIBRARY_PATH=/path/to/my/openmpi/lib:$LD_LIBRARY_PATH

which didn't help. So there is probably still some environment variable, which is not configured properly and still points to the old openmpi-installation.

[arango]

Yes, those MPI warnings are typical from gfortran. The MPI libraries are written in C and C++, allowing generic calls to enable polymorphism and type-bound objects in the argument type, calling the parallel communications function. There are specific compiling flags in gfortran for surprising such warnings, but I am very used to them.

[JMette]

You probably meant "suppressing", right? Does these warnings, by any chance can give hint at what point my configuration is incorrect?

For completeness, attached is the full output of ./build_roms.sh. The ROMS source code is under "/home/jmette/roms/" and the new open-mpi installation under "/home/jmette/openmpi/"
If someone spots something, that could help me with the right configuration I would be really thankful.

[robertson]

Are you certain that the 'mpirun' you used is at /home/jmette/openmpi/bin?

[JMette]

Yes, "which mpirun" gives the right path and also when I run it with "--prefix /home/jmette/openmpi/bin/" it shows the same error.

[JMette]

Still working on it, now back with the default mpi installation.
I've set then env variabels "PATH" and "LD_LIBRARY_PATH", which doesn't help. Is there maybe a similar env variable for the include directories, which still has to be set?

Two more questions:

Code: Select all

f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]

How/In which scripts is this path from the above build output created ? Is it in the Linux-gfortran.mk?
Because that directory exists except the openmpi/ where instead my system has an mpich/ directory. But when I switch to mpich in my build script, the path in the build output is still the same.

[robertson]

What files are in your '/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/mpich' directory?

Also, depending on the OS (redhat style vs. debian/ubuntu style), enter one of the following commands and report back what is listed.

rpm -qa | grep mpi

or:

apt list --installed | grep mpi

[JMette]

The directory contains the following:

Code: Select all

-rw-r--r-- 1 root root  14856 Feb 22  2022 mpi_base.mod
-rw-r--r-- 1 root root  29275 Feb 22  2022 mpi_c_interface_cdesc.mod
-rw-r--r-- 1 root root    865 Feb 22  2022 mpi_c_interface_glue.mod
-rw-r--r-- 1 root root  55991 Feb 22  2022 mpi_c_interface.mod
-rw-r--r-- 1 root root  24302 Feb 22  2022 mpi_c_interface_nobuf.mod
-rw-r--r-- 1 root root   3009 Feb 22  2022 mpi_c_interface_types.mod
-rw-r--r-- 1 root root  11281 Feb 22  2022 mpi_constants.mod
-rw-r--r-- 1 root root   3590 Feb 22  2022 mpi_f08_callbacks.mod
-rw-r--r-- 1 root root  16781 Feb 22  2022 mpi_f08_compile_constants.mod
-rw-r--r-- 1 root root   4796 Feb 22  2022 mpi_f08_link_constants.mod
-rw-r--r-- 1 root root 176478 Feb 22  2022 mpi_f08.mod
-rw-r--r-- 1 root root  10089 Feb 22  2022 mpi_f08_types.mod
-rw-r--r-- 1 root root  27503 Feb 22  2022 mpi.mod
-rw-r--r-- 1 root root   1234 Feb 22  2022 mpi_sizeofs.mod
-rw-r--r-- 1 root root 100119 Feb 22  2022 pmpi_f08.mod

The command "apt list --installed | grep mpi" gives:

Code: Select all

libfftw3-mpi3/jammy,now 3.3.8-2ubuntu8 amd64 [installed,automatic]
libhdf5-openmpi-103-1/jammy,now 1.10.7+repack-4ubuntu2 amd64 [installed,automatic]
libhdf5-openmpi-hl-100/jammy,now 1.10.7+repack-4ubuntu2 amd64 [installed,automatic]
libmpich-dev/jammy,now 4.0-3 amd64 [installed,automatic]
libmpich12/jammy,now 4.0-3 amd64 [installed,automatic]
libnetcdf-mpi-19/jammy,now 1:4.8.1-2 amd64 [installed]
libnetcdf-mpi-dev/jammy,now 1:4.8.1-2 amd64 [installed]
libopenmpi3/jammy,now 4.1.2-2ubuntu1 amd64 [installed,automatic]
libscalapack-openmpi2.1/jammy,now 2.1.0-4 amd64 [installed,automatic]
mpi-default-bin/jammy,now 1.14 amd64 [installed,automatic]
mpich/jammy,now 4.0-3 amd64 [installed]
openmpi-bin/jammy,now 4.1.2-2ubuntu1 amd64 [installed,automatic]
openmpi-common/jammy,now 4.1.2-2ubuntu1 all [installed,automatic]

[robertson]

Try having your sysadmin install libopenmpi-dev. It looks like the mpif.h you're looking for is contained in that package. It seems that Ubuntu requires the -dev package in order to actually compile with openmpi.

[JMette]

Reporting back:
Installing "libopenmpi-dev" really did the trick. Also without setting any custom paths I was able to run the upwelling test case. For documentation, the Linux version I was working on was Ubuntu 22.04.4 LTS.

Thanks a lot to everyone who contributed here.