compiling 4DVar with WET_DRY + wclock_off problem

[jivica]

Does anybody run or even have been able to compile the latest ROMS 4DVar for application with WET_DRY option switched on?
I fixed some easy picks in tl_zeta.F, ad_zeta.F where we need to define eps (missing definition as local variable).
Then in ad_initial.F and tl_initial.F when calling initrc or Tindex as arrays (ng) while they are defined as index.

Unfortunately, seems that I have another totally unrelated problem within distributed.F where, according to CRAY intel compiler,
libUTIL.a(timers.o): in function `wclock_off_':
timers.f90:(.text+0x304b): undefined reference to `mp_collect_'
Not sure where this one is coming from? Is is possible to turn off PROFILING?

The latest version is so much different from the one I used few months before.
It would be helpful to have suite of tests (i.e. 4DVar, WC13) compiling well before making commit to current trunk (that way I could know it is problem on my side).

Thanks,
Ivica

[arango]

Oh my goodness, no. We cannot use WET_DRY in any of the adjoint-based algorithms including any of the 4D-Var drivers. That's the reason that you will find WET_DRY_NOT_YET in the code. Every time that you see a multiplication of a state variable with a mask array, we will need another term, which is missing, because it is a quadratic state term in the variational calculus sense. Notice that in wetting and drying, the land/mask arrays are time-dependent and adjointable because of their values change at every barotropic timestep They also depend on adjointable state variables zeta, ubar, and vbar. The wetting and drying variables are not computed from first principles (i.e., a governing equation) but a coastal engineering numerical approach. Therefore, such variables are not continuously differentiable once neither twice (Hessian). Even if we differentiate the code line-by-line using variational calculus formulas to derive the tangent linear kernel, we need to save the forward basic state trajectory at every barotropic timestep, so other high-order terms (mostly quadratic) can be computed correctly. In other words, we need to have the land/sea masking trajectory history forward and backward. How you interpolate such masking arrays if their nonlinear trajectory is saved at specified snapshots? It is too much problem for such a little or no gain.

If you want to explore data assimilation with wetting and drying, maybe you should consider any of the ensemble Kalman Filters. ROMS is already connected to DART. We submitted for publication a paper comparing 4D-Var and EAKF recently. Anyway, what kind of observations will be assimilated in the wetting and drying areas? How to formulate an error covariance hypothesis in such areas?

You need to give more details about the wclock_off problem. I have revised that logic recently. The PROFILE option can be turned off in globaldefs.h.

Yes, the 4D-Var algorithms are evolving. I am working on the multiple executable 4D-Var drivers to do all kinds of applications. The drivers are split in three 4D-Var phases: background, increment, and analysis. Each phase can be computed with different executable, numerical precision, and grid resolution. The background phase can be part of a coupled system and or contain nested grids. The increment can be computed with a coarser grid and or lower precision containing the background grid. Such 4D-Var increment can be interpolated back to the finer grid and then run the analysis phase.

We are also working on the very complex saddle-point 4D-Var, which parallelizes the inner loops, so the iteractions are done concurrently to accelerate the minimization. That's the reason that you see progressive changes in the 4D-Var algorithms. All the tutorials and test repositories have changed accordingly. I always run the WC13 application before I load such changes to the repository. This business is quite complicated. So, I don't know what you are talking about untested commits.

[jivica]

Hi Hernan,
thanks for the swift reply!
Actually, what I am doing is running ROMS in super tidal region with tidal sea level variations due to tides having strong impact on stability and quality of results. In regions that I am interested in I have > 50% internal tidal energy (of total) and running model without tides is not an option. I really like approach you are going, with different model setup run for different parts of DA, and in my case I would run NLM + WET_DRY but AD and TL could run without, as you said. I would not assimilating anything in the regions where I have drying/wetting and changing mask (those are way too close to the coast anyway), however I would assimilate "raw" data in the deeper parts of domain and exploiting "4" in 4D-Var.
My impression - wrong, was that WET_DRY mask was used in all models to mask only, not for assimilating obs in those regions. I was mislead with the latest code having WET_DRY parts in ad_zeta, tl_zeta, ad_initial, tl_initial etc. This was added recently (I believe), and I was hoping you are working on it and have it done.

Regarding PROFILING, I will play with it (possibly it will speed up a code a bit). Strange thing happen with the latest pull I made, I couldn't compile even NLM model as compiler was throwing error as I wrote. Will try pulling again today and redo.

Thanks for feedback!
Ivica

[fransUgent]

Hi all,

I pulled the latest ROMS version this morning (29 May)

I can attest to the same error (for the WC13 RBL4DVAR case) NOT including WET_DRY. I picked up this problem with my own project today:

/WC13/RBL4DVAR/Build/libUTIL.a(timers.o): In function `wclock_off_':
timers.f90:(.text+0x1810): undefined reference to `mp_collect_'
timers.f90:(.text+0x1c0a): undefined reference to `mp_collect_'
collect2: error: ld returned 1 exit status

Doing a diff (with an older build) I see that in timers.f90:
190: CALL mp_collect (ng, model, nPETs, Tspv, Tend, MyCOMM) and following the breadcrumbs:

Utilities/timers.F
# ifdef ROMS_STDOUT
USE distribute_mod, ONLY : mp_collect
# endif
is included in the new version of timers.F

Obviously only if ROMS_STOUT is defined then the call to mp_collect will work.

Conversely, if CALL mp_collect (ng, model, nPETs, Tspv, Tend, MyCOMM) is included under the ROMS_STOUT header (instead of under the DISTRIBUTE) then this would solve the problem ?

I compiled with ROMS_STOUT defined and it compiled for the test case.
Should ROMS_STOUT be defined in all cases or is it a code bug?

Regards

Frans van Eeden

[arango]

Yes, that's a small bug, thank you. I am running nowadays with the ROMS_STDOUT option that generates the output file log.roms instead of the standard Fortran output from unit 6. My bad, I forgot to run without the ROMS_STDOUT. Having this option activated is very convenient when running coupled systems. Otherwise, the standard output log file will have a report soup from all models in the coupled system, and it is hard to find quick information. Some models out there have a very messy standard output information. I have very rigorous reports in ROMS due to its complexity.

Please update.