ROMS blowing up problem

[han]

Hi,

I ran ROMS with a blowing-up problem.

Found Error: 01 Line: 298 Source: ROMS/Nonlinear/main3d.F
Found Error: 01 Line: 298 Source: ROMS/Drivers/nl_ocean.h

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=2,2) in record = 0000003

Does anyone know why this happen? I checked the main3d.F, it gives:

! Compute horizontal mass fluxes (Hz*u/n and Hz*v/m), density related
! quatities and report global diagnostics.
!-----------------------------------------------------------------------
!
DO ig=1,GridsInLayer(nl)
ng=GridNumber(ig,nl)
DO tile=first_tile(ng),last_tile(ng),+1
CALL set_massflux (ng, tile, iNLM)
# ifndef TS_FIXED
CALL rho_eos (ng, tile, iNLM)
# endif
CALL diag (ng, tile)
# ifdef TLM_CHECK
CALL nl_dotproduct (ng, tile, Lnew(ng))
# endif
END DO
!$OMP BARRIER
END DO
IF (FoundError(exit_flag, NoError, __LINE__, &
& __FILE__)) RETURN

[kate]

It is from that call to diag, which checks the status of some model fields. If it finds things have gone out of bounds, it dies like that. You should look at that third record of the restart file to see what is going bad.

[han]

It is from that call to diag, which checks the status of some model fields. If it finds things have gone out of bounds, it dies like that. You should look at that third record of the restart file to see what is going bad.

Yes, I found that the velocity went bad and exceeded 20 m/s.

Thanks, kate!

[ElisaSmz]

Hi,

I have exactly the same problem : my model blows up after a few time steps and I found with the restart file than the velocity is around 20 m/s in some points. This problem appeared when I used a 1/8° resolution instead of a 1/4° resolution.
So my question is : where does it come from ? And what did you do to fix this problem ? I tried to reduce dt and it works but only for dt lower than 10 which is not convenient for a long simulation. So I decided to run with dt=10 for a few days and then use the restart file to launch another simulation with a longer time step but the model still blows up.

Thank you in advance for your help.

[kate]

Have you looked to see where things go bad? It might help you figure out how to get a longer timestep.

[mdessert]

Have you checked the value of your CFL criteria?

Minimum barotropic Courant Number = 1.17951147E-02
Maximum barotropic Courant Number = 4.07452685E-01

in the print you can read as your model is running (and that I put in a 'log_file').
If your Courant Number is too high, that means your grid can be in adapted. If you increased your horizontal resolution, you have to decrease your time step (That's information that Courant Number can provide). Some clues can be find there : viewtopic.php?f=14&t=4560&p=17748&hilit ... ber#p17748

Do you know where you're velocity became too high? If it appends on the sea floor (as it did for me), that may your bathymetry which is no more adapted to your new resolution. Do you see angular topography as you check your grid file? Maybe you should smooth it more ? I think that rx0 criteria can inform you on that point (in the print of your model). I think that it depicts a relation between your horizontal, your vertical resolution and your bathymetry (is your bottom rough or steep ? ).
Mine looks that :

Maximum grid stiffness ratios: rx0 = 1.000000E-02 (Beckmann and Haidvogel)
rx1 = 2.168223E+00 (Haney)

I've read that a rx0<0.1 was good enough. In my case, as my bathymetry was not enough accurated (lower resolution)I had to smooth it a lot... And it solves my blowing up problem !

[ElisaSmz]

First, thank you for your answers !

The velocity seem to be go bad on the boundaries (from the bottom to the surface), I suspect my tidal forcing to be at the root of the problem, but I don’t really know how.
For the CFL criteria, I have :
Minimum barotropic Courant Number = 7.50545369E-03
Maximum barotropic Courant Number = 1.50438817E-01

For the bathymetry, I tried two :
rx0 = 2.500000E-01 (Beckmann and Haidvogel)
rx1 = 1.023680E+01 (Haney)
And another one more smooth :
rx0 = 1.000000E-01 (Beckmann and Haidvogel)
rx1 = 4.292327E+00 (Haney)
The model blows up for both of them. But your bathymetry seems to be even smoother so maybe I should try to smooth my bathymetry even more to see what happens.

Thank you again for your help !

[mdessert]

Actually, I have a similar problem : my velocities, temperature (my ocean will boil soon !!!) and salinity behave abnormal near the boundaries (although the boundaries seem OK !?!)
You wrote :

The velocity seem to be go bad on the boundaries (from the bottom to the surface), I suspect my tidal forcing to be at the root of the problem, but I don’t really know how.

Have you tried without your tide forcing (just to verify whether your problem really comes from that...)? What if you deactivated them?
I did not yet activate the tide but only the open lateral boundaries, and I see some similar problems. I have temperature, salt and velocity oscillating from the boundaries to the inner domain. I've tried to add a sponge layer (with increasing diffusivity and viscosity). But that does not seem enough...

Do you use sponge layer?

[ElisaSmz]

Yes, I set up also a sponge in all boundaries with visc_factor and diff_factor up to 10. Maybe we should increase these 2 factors...

And I tried to remove tidal forcing but now, I don't know why, it's worst ! When I start my simulation, I have the following error :

forrtl: severe (64): input conversion error, unit -5, file Internal Formatted Read
27: Image PC Routine Line Source
27: oceanM 000000000085CAD6 for__io_return Unknown Unknown
27: oceanM 000000000087E2BF for_read_int_fmt_ Unknown Unknown
27: oceanM 0000000000839F76 Unknown Unknown Unknown
27: oceanM 0000000000705D6C Unknown Unknown Unknown
27: oceanM 00000000007029A0 Unknown Unknown Unknown
27: oceanM 000000000051799C Unknown Unknown Unknown
27: oceanM 000000000040C7BE Unknown Unknown Unknown
27: oceanM 000000000040BED8 Unknown Unknown Unknown
27: oceanM 000000000040BC9E Unknown Unknown Unknown
27: libc-2.17.so 00002B52D61D6B35 __libc_start_main Unknown Unknown
27: oceanM 000000000040BBA9 Unknown Unknown Unknown

and my log file stops during the reading of input files (apparently just after the reading of the initialisation file whereas I didn't change nothing in my input files !)

[kate]

You can find out what files are being referred to if you recompile with USE_DEBUG. You should get something like what is shown in the wiki.

I have never found a sponge to be helpful at the boundaries. I don't think you need the smoother bathymetry unless the steepest bits are at the boundary and where the boundary trouble is. What are your boundary conditions? Are you using fields from some other source like a global model run? You might want to try that if not.

[ElisaSmz]

Sorry for the late reply, fix all my errors took me a long time ! I did as you advised me, I turned on USE_DEBUG and I finally found where my errors came from, thank you ! If someone has the same problem I can advice first to update the varinfo.dat file and then, my main error was in my forcing files, my time unit wasn't the same than in the other files (it was 'seconds since 1948-01-01 00:00:0.0' instead of 'seconds since 1948-01-01 00:00:00') which leaded to the following error "Error: forrtl: severe (64): input conversion error, unit -5, file Internal Formatted".

And for my blowing up problem it seems to work also, I just changed the viscosity and the diffusion (I set TNU2=50 instead of TNU2=300 and I set VISC4=-4.0d9 instead of VISC4=-3.0d11) and now I can set dt=150 without any problem.

[uranoscopus]

Sorry for the late reply, fix all my errors took me a long time ! I did as you advised me, I turned on USE_DEBUG and I finally found where my errors came from, thank you ! If someone has the same problem I can advice first to update the varinfo.dat file and then, my main error was in my forcing files, my time unit wasn't the same than in the other files (it was 'seconds since 1948-01-01 00:00:0.0' instead of 'seconds since 1948-01-01 00:00:00') which leaded to the following error "Error: forrtl: severe (64): input conversion error, unit -5, file Internal Formatted".

And for my blowing up problem it seems to work also, I just changed the viscosity and the diffusion (I set TNU2=50 instead of TNU2=300 and I set VISC4=-4.0d9 instead of VISC4=-3.0d11) and now I can set dt=150 without any problem.

Hi, I have a similar blowup issue, appearing only when I put some boundaries (closed domain works fine). Did you experience the same?

UPDATE:
Solved, there was an issue in the boundaries I did'nt realize before. Just bad values.

[19zsheng]

Dear uranoscopus,

I have had similar problems, My model blew up at 10 days, apparently due to high water velocity values in historical files near the North and South border.

Have you solved this problem? Could you give me some advice?

Thanks in advance.
Best regards.
zsheng.

[kate]

I'll say what I always say: try a shorter timestep and look more closely at your solution to see where it is blowing up.

[manhnt181]

Hi Kate,
I had the same problem during running period, my model died after 2 hours forecast.
My error is displayed in the below:

Code: Select all

 DIAG speed trouble    20.0066825983225     
 Found Error: 01   Line: 351      Source: ROMS/Nonlinear/main3d.F
 Found Error: 01   Line: 332      Source: ROMS/Drivers/nl_ocean.h

 Blowing-up: Saving latest model state into  RESTART file
     REASON: MaxSpeed =  2.0E+01

It means that my max speed reach to 20.0066825983225.
How can I do to fix this problem?
Thank you very much!

[kate]

Same answer as above - use a shorter timestep and look more closely at your model output files.

[manhnt181]

Hi Kate,
I have plotted some valuables in the last hst output file but I didn't see somethings went bad in my case or maybe with my limited knowledge about the ROMS model I didn't see that.
Can you kindly help me to find the solution for this blow-up error?
I have tried to use a shorter timestep but it still occurred.
Thank you very much!

[kate]

How long before the final blow-up is this snapshot? You are running COAWST? I have no experience with modes of blowing up for anything but the ocean component.

For the ocean, it can go from fine to bad in the blink of an eye. A history snapshot some hours before the trouble can look great. I tend to look through the restart record it saves as it's blowing up. Did the velocity get large? Is it u or v or both? It is often just the one, at mid-depth. Looking at surface plots tells you nothing.

[manhnt181]

HI Kate,
Thank you very much for your reply.
My blow-up issue came up after more than 09 hours running as you can see here:

Code: Select all

Timing for main: time 2017-11-02_21:15:20 on domain   2:    1.68971 elapsed seconds
      2230 2017-11-02 21:17:30.00  8.227072E-02  2.282634E+04  2.282642E+04  8.107080E+16
                     (216,070,02)  1.344279E-04  1.020814E-05  5.272707E-01  2.004695E+01
 DIAG speed trouble    20.0469512008091     
 Found Error: 01   Line: 341      Source: ROMS/Nonlinear/main3d.F
 Found Error: 01   Line: 331      Source: ROMS/Drivers/nl_ocean.h, ROMS_run

 Blowing-up: Saving latest model state into  RESTART file
     REASON: MaxSpeed =  2.0E+01

Timing for main: time 2017-11-02_21:15:40 on domain   2:    1.70914 elapsed seconds
      WRT_RST     - wrote re-start    fields (Index=1,1) in record = 3

I also displayed u, v momentum component in all of my vertical levels, however I didn't find any abnormal value.
Can you kindly help me to find out the solution here?
I really appreciated your help.

[kate]

Those NE boundary conditions look ghastly. Something is going on there, I don't know what.