Beware of netCDF STRINGs

[m.hadfield]

This isn't actually a ROMS bug at all (or even a picture of one) but for 6 hours on Wednesday I thought it was.

I ran into a problem where ROMS was not reading the units attribute of a time variable in an input netCDF file. In fact it wasn't correctly reading any string attributes at all.

The files were generated in IDL, as is my wont. I had changed my file format to NETCDF4. It turns out that when you add an attribute containing IDL string data to a NETCDF4 file, it stores it with the new-fangled STRING data type, rather than the traditional CHAR data type. (The CHAR type supports fixed-length strings as character arrays and is the only one available in the NETCDF3 format. The STRING type supports variable-length strings. Quite why any of this matters for an attributes containing a single string is not clear to me.) The netCDF Fortran-90 interface cannot read or write STRINGs. The fix was to force the format to be CHAR.

The gory details can be found here:

http://www.unidata.ucar.edu/mailing_lis ... 00100.html
http://www.unidata.ucar.edu/mailing_lis ... 00103.html
https://bugtracking.unidata.ucar.edu/br ... FORTRAN-29

Let that be a warning to you.

[arango]

Yes, I read your postings in the NetCDF forum. This is an annoying one. I always use the CHAR flags. Characters and strings are awkward in NetCDF. I am pretty conservative about the new features in the NetCDF library. I struggled a lot with them when designing the NetCDF file for the nesting contact points. The structures of the NetCDF-4 library were not robust so I decided to pack/write the contact point data with the elementary functions. It is a little complicated but robust. I just needed to write unpacking routines for Matlab and ROMS.

[angelolemos]

Hi,

I am using the Roms Agrif version in an interannual experiment (October/2015).
Using data from HYCOM to generate CLM file, I modified the create_clim to avoid large netcdf file.
I modified the code of create_climfile.m file:
from:
ncid=netcdf.create(clmname,'64BIT_OFFSET');
to:
ncid=netcdf.create(clmname,'NETCDF4');

Now, I have the following file:
roms_clm_HYCOM_Y2015M10.nc 34GB

Finally, when I try to run the model, a problem appear:
./jobcomp
./run_roms_inter.csh
.
.
.
Time splitting: ndtfast = 30 nfast = 44


Maximum grid stiffness ratios: rx0 =0.2321427218 rx1 = 4.3590976899

GET_INITIAL -- Processing data for time = 273.0 record = 1

GET_TCLIMA - unable to open climatology NetCDF file: ../SCRATCH/roms_clm.nc

MAIN - number of records written into history file(s): 0
number of records written into restart file(s): 0
number of records written into averages file(s): 0


ERROR: Abnormal termination: netCDF INPUT

My Cu_max is 0.46.
I think that the error is because the size of same variables of the CLM file, i.e., u, v, temp and salt > 4GB.
I checked the data from CLM file and everything is ok.
Someone can help?

[kate]

* You should start a new thread for this sort of issue.
* You should probably be posting on a ROMS_AGRIF forum, not here.
* If you were using the myroms ROMS, you could split your climatology file into smaller chunks (by time, at least), so that you could have say one year per file, then string the files together in the ocean.in file.
* If you can convince the ROMS_AGRIF people to be more flexible in their forcing/BC/CLM inputs, more of us would be willing to try it out.

[shchepet]

Kate (and Angelolemos as well),

You should be more specific here: rather than advise to post it on AGRIF forum rather than
here, or trying to motivate AGRIF people to be more flexible (good luck with this, huh...)

The error

GET_TCLIMA - unable to open climatology NetCDF file: ../SCRATCH/roms_clm.nc

is 99.99% likely caused not by what kind of branch of ROMS code is used, but by a simple fact
that netCDF library used during compilation of ROMS executable is not netCDF-4 capable, thus
cannot process this kind of file (hence returns ERROR status when attempting to open such file)

As a matter of fact, when opening a netCDF file in read-only mode from a Fortran or C program
there is no difference whatsoever between netCDF-3 and netCDF-4 in terms of user-level program:
all operations associated with handling HDF-5 (a.k.a. netCDF-4) format, including identification
of what type of file it us and taking appropriate logical branching are done internally within
netCDF library itself and generally requires no changes in user Fortran code.

So the key question Angelolemos (you should also be more specific): did you compile your
executable using netCDF library compiled with netCDF-4 enabled?

If the answer is "no" then recompile your netCDF library and make sure that netCDF-4
capabilities work, then recompile your code and see whether the problem remains.

[shchepet]

Now, I have the following file: roms_clm_HYCOM_Y2015M10.nc 34GB

Another thing to mention here is that 64BIT_OFFSET netCDF format has virtually no restriction
of the file size, however to exceed 2 GByte limit the file MUST HAVE UNLIMITED DIMENSION, and

(1) all variables not having unlimited dimension put together must not exceed 2 GBytes.

(2) the combined size of one record of all variables having unlimited dimension must not exceed 2 GBytes.

In practice, given the size of practical grids in ROMS simulations the above conditions (1)
and (2) can always be met if unlimited dimension is reserved for time records.

So when you create netCDF file from Matlab by ncid=netcdf.create(clmname, ...) followed by
defining dimensions did you make unlimited dimension for time or have it fixed-size?

If fixed, make it unlimited and do not bother with netCDF-4. Then everything should work as is.

[kate]

Sorry, just feeling grumpy about the state of information flow between ROMS branches.

I'm using climatology files which are 14 GB, one file per year, no problem.