Can't run my IS4D-Var case Gnorm = NaN

[Arvin_Yin]

Hi, everyone!

When I run my IS4D-Var model, it blows up at the beginning of the first inner loop. The log file shows:
…
<< Descent Algorithm >
(001,000): Initial gradient norm, Gnorm = NaN
…
< 4D Variational Data Assimilation, Outer = 001, Inner = 001 >
STEP KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 NaN NaN NaN 1.287561E+16

The initial gradient norm is NaN and I think this is why the model blows up. But I don’t know why the model outputs initial gradient norms of NaN.

My ocean.in and s4dvar.in file are like those in WC13 case. At first I ran the forward model for one year. Then I computed the standard deviations using this forward model output data and wrote them into the initial STD files(use the ROMS matlab script d_std.m). The normalization coefficients were computed using the exact method. The observation data are AVISO SLA data and NOAA OISST data. I added the mean sea surface, which are calculated from the one year forward model, to the AVISO SLA data, then wrote them into the observation file(use ROMS matlab script d_ssh_obs.m, d_sst_obs.m and obs_merge.m).

Also, I noticed that, when the model only includes the SLA data or the SST data in the observation file, the model rejected most of them, but if the model includes both the SLA data and the SST data, all the observation data can be accepted by the model.

I attached my log file for your convenience.

Thank you for your help.

[susonic]

Hi,
Does your domain include equator?
If it's ok for you, I want to see your model domain.

-JH

[Arvin_Yin]

Hi,
Does your domain include equator?
If it's ok for you, I want to see your model domain.

-JH

Hi, JH
Thanks.
My domain doesn't include equator.It's 16N - 24N, 105E - 132E, in north South China Sea.

And I have another question while I compile with SANITY_CHECK, the error goes:

cd /mnt/volume04/yinpeng/Projects/nSCS/Build; /usr/local/mpich2-1.2.1p1/bin/mpif90 -c -frepack-arrays -O3 -ffast-math ocean_control.f90
ocean_control.f90:394.58:

CALL mp_collect (ng, iTLM, 4, IniVal, val(1,ng))
1
Error: There is no specific subroutine for the generic 'mp_collect' at (1)
make: *** [/mnt/volume04/yinpeng/Projects/nSCS/Build/ocean_control.o] Error 1

In fact I'm not quite clear ahout what deos SANITY_CHECK do.But have you ever met this error?

Thank you again for your help.

[susonic]

I've never done SANITY_CHECK before. But I compiled with SANITY_CHECK and faced with same error message as you noted above(with gfortran).

Maybe code developer can helped this out to solve the problem.


I saw the log file you attached and found TLM increments are zero even though ROMS continuing the intergration. I was wondering why this value is appearing.

Would you give same value to nHIS and nOBC, nSFF(say nHIS=240, nOBC=240, NSFF=240) and run the model again?

[susonic]

Also, I noticed that, when the model only includes the SLA data or the SST data in the observation file, the model rejected most of them, but if the model includes both the SLA data and the SST data, all the observation data can be accepted by the model.

For above situation, try to use super observation file instead.

[Arvin_Yin]

Many thanks.

For above situation, try to use super observation file instead.

Yes, it works!

I saw the log file you attached and found TLM increments are zero even though ROMS continuing the intergration. I was wondering why this value is appearing.

Would you give same value to nHIS and nOBC, nSFF(say nHIS=240, nOBC=240, NSFF=240) and run the model again?

I tried NHIS=NOBC=NSFF=240(120), but the results remained the same.

[susonic]

I checked out the log file you attached several times but it is difficult to diagnose the causes of NaN value.

By the way, your Haney(rfactor) and Beckman & Haidvogel number is quiet high compare with known value(say, rfactor < 0.2) which means you did not smooth the bottom topography much.

[wavewatch]

Hi,

I wonder if you have solved this problem?
Now I have the similar error like you in my IS4D-Var run, my initial gradient norm (Gnorm) is NaN.

<<<< Descent Algorithm >>>>

(001,000): Initial gradient norm, Gnorm = NaN
……
TLM: GET_STATE - Read state initial conditions, t = 1 00:00:00
(Grid 01, File: wc13_itl.nc, Rec=0001, Index=1)
- free-surface
(Min = NaN Max = NaN )
- free-surface, open boundaries
(Min = NaN Max = NaN )
- vertically integrated u-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated u-momentum component, open boundaries
(Min = NaN Max = NaN )
- vertically integrated v-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated v-momentum component, open boundaries
(Min = NaN Max = NaN )
- surface u-momentum stress, adjusted tl_ustr
(Min = NaN Max = NaN )

……

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 1 00:00:00 NaN NaN NaN 1.159856E+16

Could you tell me more about this error and how you solved it?

Thank you very much.

[ckerry]

Hi,
I'm also wondering if you solved this problem and I am having the same problem.

<<<< Descent Algorithm >>>>

(001,000): Initial gradient norm, Gnorm = NaN

.....

TLM: GET_STATE - Read state initial conditions, t = 4367 00:00:00
(File: ocean_itl.nc, Rec=0001, Index=1)
- free-surface
(Min = NaN Max = NaN )
- free-surface, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated u-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated u-momentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated v-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated v-momentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- surface u-momentum stress, adjusted tl_ustr
(Min = NaN Max = NaN )
- surface v-momentum stress, adjusted tl_vstr
(Min = NaN Max = NaN )
- u-momentum component
(Min = NaN Max = NaN )
- u-momentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- v-momentum component
(Min = NaN Max = NaN )
- v-momentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- potential temperature
(Min = NaN Max = NaN )
- salinity
(Min = NaN Max = NaN )
- potential temperature, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- salinity, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- surface net heat flux, adjusted tl_tflux
(Min = NaN Max = NaN )
- surface net salt flux, (E-P)*SALT, adjusted tl_tflux
(Min = NaN Max = NaN )

....

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME

0 4367 00:00:00 NaN NaN NaN 1.188685E+14

Blowing-up: Saving latest model state into RESTART file