ROMS whit SWAN problem

[mistral]

Hello everyone

I tried to run intel_test on a cluster, with MPI and gfortran.
I compiled the libraries MCT (. / configure, make, with this Makefile.conf)
makefile.conf:


# COMPILER, LIBRARY, AND MACHINE MAKE VARIABLES

# FORTRAN COMPILER VARIABLES #

# FORTRAN COMPILER COMMAND
FC = mpif90

# FORTRAN AND FORTRAN90 COMPILER FLAGS
FCFLAGS = -O2

# FORTRAN90 ONLY COMPILER FLAGS
F90FLAGS = -w

# FORTRAN COMPILE FLAG FOR AUTOPROMOTION
# OF NATIVE REAL TO 8 BIT REAL
REAL8 = -r8

# FORTRAN COMPILE FLAG FOR CHANGING BYTE ORDERING
ENDIAN = -convert big_endian

# INCLUDE FLAG FOR LOCATING MODULES (-I, -M, or -p)
INCFLAG = -I

# INCLUDE PATHS (PREPEND INCLUDE FLAGS -I, -M or -p)
INCPATH =

# MPI LIBRARIES (USUALLY -lmpi)
MPILIBS =

# PREPROCESSOR VARIABLES #

# COMPILER AND OS DEFINE FLAGS
DEFS = -DSYSLINUX -DCPRINTEL

# SET A SEPARATE PREPROCESSOR COMMAND IF FORTRAN COMPILER
# DOES NOT HANDLE PREPROCESSING OF SOURCE FILES

# FORTRAN PREPROCESSOR COMMAND
FPP = cpp
# FOTRAN PREPROCESSOR FLAGS
FPPFLAGS = -P -C -N -traditional

# C COMPILER VARIABLES #

# C COMPILER
CC = cc

# C COMPILER FLAGS - APPEND CFLAGS
ALLCFLAGS = -DFORTRAN_UNDERSCORE_ -DSYSLINUX -DCPRINTEL -O

# LIBRARY SPECIFIC VARIABLES #

# USED BY MCT BABEL BINDINGS
COMPILER_ROOT =
BABELROOT =
PYTHON =
PYTHONOPTS =

# USED BY MPI-SERIAL LIBRARY

# SIZE OF FORTRAN REAL AND DOUBLE
FORT_SIZE =

# SOURCE FILE TARGETS #

# ENABLE RULE FOR PROCESSING C SOURCE FILES
# BY SETTING TO .c.o
CRULE = .c.o

# ENABLE RULE FOR PROCESSING F90 SOURCE FILES
# BY SETTING TO .F90.o AND DISABLING F90RULECPP
F90RULE = .F90.o

# ENABLE RULE FOR PROCESSING F90 SOURCE FILES
# WITH SEPARATE PREPROCESSING INVOCATION
# BY SETTING TO .F90.o AND DISABLING F90RULE
F90RULECPP = .F90RULECPP

# INSTALLATION VARIABLES #

# INSTALL COMMANDS
INSTALL = /home/clima/a/ROMSanto/MCT/install-sh -c
MKINSTALLDIRS = /home/clima/a/ROMSanto/MCT/mkinstalldirs

# INSTALLATION DIRECTORIES
abs_top_builddir= /home/clima/a/ROMSanto/MCT
MCTPATH = /home/clima/a/ROMSanto/MCT/mct
MPEUPATH = /home/clima/a/ROMSanto/MCT/mpeu
EXAMPLEPATH = /home/clima/a/ROMSanto/MCT/examples
MPISERPATH =
libdir = /usr/local/lib
includedir = /usr/local/include

# OTHER COMMANDS #
AR = ar cq
RM = rm -f





I edited LINUX-ifort.mk with these lines

ifdef USE_MCT
# MCT_INCDIR ?= /
# MCT_LIBDIR ?= /
# FFLAGS += -I$(MCT_INCDIR)
# LIBS += -L$(MCT_LIBDIR) -lmct -lmpeu

FFLAGS += -FIXED -I$/home/clima/a/ROMSanto/MCT/mct -I$/home/clima/a/ROMSanto/MCT/mpeu
MCT_LIBDIR ?= /home/clima/a/ROMSanto/MCT/mct
MPEU_LIBDIR ?=/home/clima/a/ROMSanto/MCT/mpeu
LIBS += -L$(MCT_LIBDIR) -lmct -L$(MPEU_LIBDIR) -lmpeu




but I get this error in the compilation:
cd Build; /opt/intel/ict/3.0/mpi/3.0/bin64/mpif90 -c -frepack-arrays -O3 -ffast-math -Ihome/clima/a/ROMSanto/MCT/mct -Ihome/clima/a/ROMSanto/MCT/mpeu bulk_flux.f90
waves_coupler.f90:17.72:

USE m_MCTWorld, ONLY : MCTWorld_init => init
1
Fatal Error: Can't open module file 'm_mctworld.mod' for reading at (1): No existe el fichero o el directorio
make: *** [Build/waves_coupler.o] Error 1
make: *** Se espera a que terminen otras tareas....


someone help me?!

thanks a lot

[jcwarner]

some suggestions:
1) do not edit LINUX-ifort.mk .
Instead, set your environmet with:

MCT_LIBDIR = /home/clima/a/ROMSanto/MCT/mct
MPEU_LIBDIR =/home/clima/a/ROMSanto/MCT/mpeu

(notice - i removed the '?' symbol)

2) here is the error
Fatal Error: Can't open module file 'm_mctworld.mod' for reading at (1): No existe el fichero o el directorio
So, the questionis : 'Does 'm_mctworld.mod exist in that el directorio?

[mistral]

hi jw

thanks for the fast
you think is a good idea to use gfortran???

I compiled mct but I received this error

Machine=x86_64
OS=Linux
Fortran Compiler is Intel ifort

Output Variables: {CC=cc} {ALLCFLAGS=-DFORTRAN_UNDERSCORE_ -DSYSLINUX -DCPRINTEL -O} {FPP=cpp} {FPPFLAGS=-P -C -N -traditional} {FC=mpif90} {F90=ifort} {FCFLAGS=} {F90FLAGS=-w -ftz -assume byterecl} {MACHDEFS=-DSYSLINUX -DCPRINTEL} {OPT=-O2} {REAL8=-r8} {BIT64=} {ENDIAN=-convert big_endian} {MPIF90=mpif90} {MPILIBS=} {MPIHEADER=} {INCLUDEFLAG=-I} {INCLUDEPATH=} {AR=ar cq} {BABELROOT=} {COMPILER_ROOT=} {PYTHON=} {PYTHONOPTS=} {FORT_SIZE=} {prefix=NONE}

configure: creating ./config.status
config.status: creating Makefile.conf
Please check the Makefile.conf
Have a nice day!
ROMSanto/MCT> make
make[1]: se ingresa al directorio `/home/clima/a/ROMSanto/MCT/mpeu'
cc -c -DFORTRAN_UNDERSCORE_ -DSYSLINUX -DCPRINTEL -O get_zeits.c
mpif90 -c -DSYSLINUX -DCPRINTEL -O2 -w -ftz -assume byterecl m_mpif.F90
gfortran: byterecl: No existe el fichero o el directorio
cc1: error: unrecognized command line option "-ftz"
make[1]: *** [m_mpif.o] Error 1
make[1]: se sale del directorio `/home/clima/a/ROMSanto/MCT/mpeu'
make: *** [subdirs] Error 2
ROMSanto/MCT>


makefile.conf-->




# COMPILER, LIBRARY, AND MACHINE MAKE VARIABLES
# FORTRAN COMPILER VARIABLES #
# FORTRAN COMPILER COMMAND
FC = mpif90
# FORTRAN AND FORTRAN90 COMPILER FLAGS
FCFLAGS = -O2
# FORTRAN90 ONLY COMPILER FLAGS
F90FLAGS = -w -ftz -assume byterecl
# FORTRAN COMPILE FLAG FOR AUTOPROMOTION

and I modified the text:

# FORTRAN90 ONLY COMPILER FLAGS
F90FLAGS = -w
# FORTRAN COMPILE FLAG FOR AUTOPROMOTION

and the compilation is finished with "success" ...
but the file can not find m_mctword.mod


I'm a little confused about MCT libraries...

[jcwarner]

these sound like questions to post on the MCT lib web site.
I do not use gfortran, so i can not provide guidance on fflags for that compiler.
When the MCT is built, there should be *.mod files in the /mct and /mpeu dirs. If you then type 'make install', then those *.mod files will be copied to a location that is specified in the Makefile.conf file.

[kate]

You have this:

Code: Select all

Fortran Compiler is Intel ifort

and yet you say you are using gfortran. Maybe if you can consistently use one compiler for everything - and let it know you are doing so - it will cooperate.

[mistral]

okkk resolved
thanks

I've compiled so MCT

. / configure FC = mpif90

Currently I have a new (and known) problem (I think it is caused by gfortran)



cd Build; /opt/intel/ict/3.0/mpi/3.0/bin64/mpif90 -c -frepack-arrays -O3 -ffast-math -I/home/clima/a/ROMSanto/MCT/mct -I/home/clima/a/ROMSanto/MCT/mpeu -ffixed-form swancom4.f90
ocean_coupler.f90:436.32:

IF (A(i).eq.1.0_r8/0.0_r8) THEN
1
Error: Division by zero at (1)
ocean_coupler.f90:438.17:

END IF
1
Error: Expecting END DO statement at (1)
make: *** [Build/ocean_coupler.o] Error 1
make: *** Se espera a que terminen otras tareas....
^Cmake: *** [Build/swancom1.o] Interrupción

change 0.0 to 0.1 ...?!

[mistral]

clima@perico:~/a/ROMSanto> mpirun -np 16 ./oceanM ROMS/External/coupling_test_head.in
WARNING: Can't read mpd.hosts for list of hosts, start only on current
Coupled Input File name = ROMS/External/coupling_test_head.in

Model Coupling Parallel Threads:

Ocean Model MPI nodes: 000 - 007
Waves Model MPI nodes: 008 - 015


Ocean Export: bath:SSH:Ubar:Vbar:ZO
Waves Export: Wdir:Wamp:Wlen:Wptop:Wpbot:Wubot


Process Information:

Node # 0 (pid= 8613) is active.


Node # 1 (pid= 8601) is active.


Node # 2 (pid= 8599) is active.


Node # 5 (pid= 8602) is active.


Node # 6 (pid= 8606) is active.


Node # 3 (pid= 8605) is active.


Node # 4 (pid= 8600) is active.


Node # 7 (pid= 8603) is active.
Node # 1 CPU: 0.028

Model Input Parameters: ROMS/TOMS version 3.4
Tuesday - February 16, 2010 - 1:47:21 AM
-----------------------------------------------------------------------------

Sediment Test Headland Case

Operating system : Linux
CPU/hardware : x86_64
Compiler system : gfortran
Compiler command : /opt/intel/ict/3.0/mpi/3.0/bin64/mpif90
Compiler flags : -frepack-arrays -O3 -ffast-math -I/home/clima/a/ROMSanto/MCT/mct -I/home/clima/a/ROMSanto/MCT/mpeu -ffree-form -ffree-line-length-none

Input Script : ROMS/External/ocean_test_head.in

SVN Root URL : https://www.myroms.org/svn/src/trunk
SVN Revision : exportado

Local Root : /home/clima/a/ROMSanto
Header Dir : /home/clima/a/ROMSanto/ROMS/Include
Header file : test_head.h
Analytical Dir: /home/clima/a/ROMSanto/ROMS/Functionals

Resolution, Grid 01: 0056x0026x010, Parallel Nodes: 8, Tiling: 001x002

ROMS/TOMS: Wrong choice of domain 01 partition or number of parallel threads.
Node # 3 CPU: 0.028
Node # 4 CPU: 0.028
NtileI * NtileJ must be equal to the number of parallel nodes.
Change -np value to mpirun or
change domain partition in input script.

Elapsed CPU time (seconds):

Node # 0 CPU: 0.028
Node # 2 CPU: 0.028
Node # 5 CPU: 0.028
Node # 6 CPU: 0.028
Node # 7 CPU: 0.028
*** glibc detected *** ./oceanM: free(): invalid pointer: 0x0000000000b49120 ***
*** glibc detected *** ./oceanM: malloc(): memory corruption: 0x0000000000b49120 ***
======= Backtrace: =========
======= Backtrace: =========
/lib64/libc.so.6[0x7f895cf9eaf8]
/lib64/libc.so.6[0x7fa5bd1e9af8]
/lib64/libc.so.6[0x7f895cfa14dc]
/lib64/libc.so.6(cfree+0x76)[0x7fa5bd1eb6e6]
/lib64/libc.so.6(__libc_malloc+0xa1)[0x7f895cfa3111]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpi.so.3.1(MPID_Irecv+0x206)[0x7fa5be309f86]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpi.so.3.1(MPIC_Sendrecv+0xf6)[0x7fa5be2fd98e]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpi.so.3.1(MPIDI_CH3U_Post_data_receive+0x1fa)[0x7f895e078560]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpi.so.3.1(MPIDI_CH3U_Handle_recv_pkt+0x1b51)[0x7f895e0774e1]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpi.so.3.1(MPIR_Allgather+0x73e)[0x7fa5be29d318]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpi.so.3.1(MPIDI_CH3I_SHM_read_progress+0x88)[0x7f895e071ace]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpi.so.3.1(MPIDI_CH3I_SSHM_Progress+0x7a)[0x7f895e06c0d6]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpi.so.3.1(MPI_Allgather+0x672)[0x7fa5be29b976]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpigf.so.3.1(pmpi_allgather__+0x6a)[0x7fa5be635c8a]
/opt/intel/ict/3.0/mpi/3.0/lib64/libmpi.so.3.1(MPIDI_CH3

...

[mistral]

what is the compiler that you used
and the flags for MCT??
mpif90 is not a good idea
at your suggestion I prefer to use the same compiler for all.
now I have used
for
mpich: ifort
MCT: mpif90
make: gfortran

[mistral]

1433 0 11:56:30 1.357294E-01 9.617819E+01 9.631391E+01 1.952436E+10
1434 0 11:57:00 1.352699E-01 9.617825E+01 9.631352E+01 1.952438E+10
1435 0 11:57:30 1.348071E-01 9.617832E+01 9.631313E+01 1.952439E+10
1436 0 11:58:00 1.343405E-01 9.617839E+01 9.631273E+01 1.952441E+10
1437 0 11:58:30 1.338703E-01 9.617846E+01 9.631233E+01 1.952442E+10
1438 0 11:59:00 1.333965E-01 9.617853E+01 9.631193E+01 1.952444E+10
1439 0 11:59:30 1.329191E-01 9.617860E+01 9.631152E+01 1.952445E+10
1440 0 12:00:00 1.324382E-01 9.617867E+01 9.631111E+01 1.952447E+10
OCN2WAV - (06) imported and (05) exported fields, t = 0 12:00:00
- ROMS coupling exchanges wait clock (s):
(Recv= 4.54428400E+00 Send= 4.00000000E-03)
- ROMS Import: wave direction
(Min= 4.79368996E-06 Max= 6.28318424E+00)
- ROMS Import: significant wave height
(Min= 9.88496065E-01 Max= 2.66357613E+00)
- WAV2OCN - (05) imported and (06) exported fields, t = 20030101.120000
- SWAN coupling exchanges wait clock (s):
(Recv= 1.44007000E-01 Send= 0.00000000E+00)






ROMS Import: average wave length
(Min= 3.44284821E+01 Max= 8.48340149E+01)
- ROMS Import: surface wave relative peak period
(Min= 9.51826668E+00 Max= 1.05963402E+01)
- ROMS Import: bottom wave period
(Min= 8.82794857E+00 Max= 1.03408213E+01)
- ROMS Import: wave bottom orbital velocity
(Min= 2.12755665E-01 Max= 1.07051337E+00)
- ROMS Export: bathymetry
(Min= 1.84296031E+00 Max= 2.00144238E+01)
- ROMS Export: free-surface
(Min= -9.12203553E-02 Max= 2.71493276E-02)
- ROMS Export: vertically integrated u-momentum component
(Min= -1.37517488E+00 Max= 7.66735720E-01)
- ROMS Export: vertically integrated v-momentum component
(Min= -4.06823557E-01 Max= 2.96282038E-01)
- ROMS Export: bottom roughness
(Min= 3.00 Node # 3 CPU: 258.688
000000E-03 Max= 3.00000000E-03)
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000013
WRT_RST - wrote re-start fields (Index=1,1) into time record = 0000002

Elapsed CPU time (seconds):

Node # 0 CPU: 258.688
Node # 2 CPU: 258.688
Node # 1 CPU: 258.060
At line 3791 of file swanparll.f90 (unit = 274, file = 'vel.mat-001')
Fortran runtime error: Bad value during floating point read
rank 4 in job 1 perico_54890 caused collective abort of all ranks
exit status of rank 4: return code 2
[1]+ Done mpirun -np 8 ./oceanM ROMS/External/coupling_test_head.in > run.log


I completed a run, but I received this error at the end of the run.
was never presented to you???

[dhson]

I've got similar problem:
+ On my laptop dual 2 core (ubuntu 9.10, x64, MPICH2 and every related things made from gfortran 4.3,...) my problem described in viewtopic.php?f=14&t=1743
+ Yesterday, the above problem was disappeared and I've got fantastic results when I moved my ROMS-SWAN (exactly the same) to my server (8 nodes) with the same system.
+ Today when I move my ROMS-SWAN (exactly the same) to a faster cluster but CENTOS and PGI (of course I've got to rebuild NETCDF, HDF5, MCT ), I've got:

GET_NGFLD - suspended cohesive sediment northern boundat = 90 00:00:00 class 01,
(Rec=0001, Index=1, File: bry.nc)
(Tmin= 90.0000 Tmax= 270.0000)
(Min = 1.00000000E-01 Max = 1.00000000E-01)
PGFIO-F-217/formatted read/unit=48/attempt to read past end of file.
File name = depth.mat-001 formatted, sequential access record = 44101
In source file swanparll.f90, at line number 4841
rank 7 in job 2 node1_40799 caused collective abort of all ranks
exit status of rank 7: return code 1
Fatal error in MPI_Barrier: Other MPI error, error stack:
MPI_Barrier(406)..........................: MPI_Barrier(comm=0x84000000) failed
MPIR_Barrier(77)..........................:
MPIC_Sendrecv(126)........................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_read(637)..............: connection failure (set=0,sock=3,errno=104:Connection reset by peer)[cli_7]: aborting job:
Fatal error in MPI_Barrier: Other MPI error, error stack:
MPI_Barrier(406)..........................: MPI_Barrier(comm=0x84000000) failed
MPIR_Barrier(77)..........................:
MPIC_Sendrecv(126)........................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_reFatal error in MPI_Barrier: Other MPI error, error stack:
MPI_Barrier(406)..........................: MPI_Barrier(comm=0x84000000) failed
MPIR_Barrier(77)..........................:
MPIC_Sendrecv(126)........................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_read(637)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer)[cli_6]: aborting job:
Fatal error in MPI_Barrier: Other MPI error, error stack:
MPI_Barrier(406)..........................: MPI_Barrier(comm=0x84000000) failed
MPIR_Barrier(77)..........................:
MPIC_Sendrecv(126)........................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_rerank 6 in job 2 node1_40799 caused collective abort of all ranks
exit status of rank 6: return code 1
ad(637)..............: connection failure (set=0,sock=3,errno=104:Connection reset by peer)
ad(637)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer)
Fatal error in MPI_Wait: Other MPI error, error stack:
MPI_Wait(156).............................: MPI_Wait(request=0x7fbfffe690, status0xbf3054) failed
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_read(637)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer)[cli_2]: aborting job:
Fatal error in MPI_Wait: Other MPI error, error stack:
MPI_Wait(156).............................: MPI_Wait(request=0x7fbfffe690, status0xbf3054) failed
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_read(637)..............: connection failure (set=0,sock=1,errno=104:Connection reset by peer)
rank 4 in job 2 node1_40799 caused collective abort of all ranks
exit status of rank 4: return code 1

+ But without coupling (ROMS, SEDIMENT, PSOURCE, WET_DRY...), ROMS run normally in all three system

MY FEELING is there is still bugs in coupling procedures or in SWAN.

[LiuZHQ]

I got the same problem say "*** glibc detected *** ./oceanM: free(): invalid next size (normal): 0x00000000352ae2f0 ***
"
and I skip the new version of ROMS and back to the ROMS148, and the problem disappeared. This is a possible solution before anyone find out the real reason.

[mistral]

first problem

I solved by setting consistently the number of cpus for each model , number of node and the number of tile partitions to use, remembering that "the product of NtileI and NtileJ
! must be EQUAL to the number of parallel nodes specified during
! execution with the "mprun" or "mpirun" command."

in my case I started by setting 2 CPU (1 swan - 1 ROMS , 2 * 1 tile part. , mpirun-np 2 ...



for the second proble I fixed by setting on a consistent time to run.
1 day for Swan = 1 day for ROMS
in fact the error appears when here one of the two models ends the run and the other is still running .



best regards

[jafar1979]

dear
i have installed MCT source only (with out data)and put its include and lib directories in linux_gfortran.mk file as:

Code: Select all

MCT_INCDIR?=/home/<user>/MCT/include
MCT_LIBDIR?=/home/<user>/MCT/lib
FFLAGS+=-I$(MCT_INCDIR)
LIB+=-L$(MCT_LIBDIR) -lmct -lmpeu

(2 lat line with out change)
but in above post i see that mistarl change linux-ifort.mk as:

FFLAGS += -FIXED -I$/home/clima/a/ROMSanto/MCT/mct -I$/home/clima/a/ROMSanto/MCT/mpeu
MCT_LIBDIR ?= /home/clima/a/ROMSanto/MCT/mct
MPEU_LIBDIR ?=/home/clima/a/ROMSanto/MCT/mpeu
LIBS += -L$(MCT_LIBDIR) -lmct -L$(MPEU_LIBDIR) -lmpeu

i need to know that my changes are enough? or i should work as mistral
cheers
jafar