[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20040818: Very large netCDF files

"Tom Black" <address@hidden> writes:

> Ed and Russ,
> 
> I want to thank you both for your quick repsonses.  We are very
> inexperienced with the use of netCDF here at NCEP since our operations
> do not currently support it (we use strictly machine binary, GriB, and
> BUFR).  In our collaboration with NCAR and FSL on the Weather and
> Research Forecast (WRF) effort, we have needed to begin making use of
> netCDF.  Our current problem arises in a rather large, complicated job
> called SI (Standard Initialization) that originated at FSL which reads
> in and digests data and then writes out files that are themselves used
> as input for a final step that produces the initial and boundary
> conditions for our WRF forecasts.  Obviously the larger the domain and
> the denser the gridpoint spacing we use, the larger the files are that
> the SI has to produce.  When their size exceeds something around 2
> gigabytes, they are corrupted.  We currently have an urgent need to
> run a large domain, high resolution forecast but we are unable to
> create those initial and boundary conditions because of the SI files.
> 
> First, let me say again that we are pretty ignorant in using netCDF.
> So Ed, just where do I look to see the nc_create that you mentioned
> when the files are being written out and where is NC_64BIT_OFFSET
> specified?

When you create the netcdf file, you do so with the nc_create call.

The second parameter is the cmode, and in order to get a large file,
you need to logically or NC_64BIT_OFFSET, like this:

   stat = nc_create("rec2Gp.nc", NC_CLOBBER|NC_64BIT_OFFSET, &ncid);

This call opens a file called rec2Gp.nc, overwritting any existing
file of that name (that's what NC_CLOBBER means) and making rec2Gp.nc
a 64-bit offset file.

The version of netcdf that you are using contains a significant
bug. It will not tell you if you exceed size limits, it will just
silently mess up. The next beta, and the next release of netcdf, both
return error codes if you try and create a file that is too large.

> 
> Russ, it would seem impractical for me to transfer all the SI
> directories and codes (there are many) and input data files (also very
> many different ones) for our very large run for you to try to run it
> yourself.  There are still significant parts of it that we must simply
> run as black boxes so I can't explain alot of the guts of it to you.
> Let me know what you think about this.
> 
> Again, we really appreciate your willingness to help us out with this
> critical problem.

Can you tell me what platform you built netcdf on?


> 
> Tom
> 
> Ed Hartnett wrote:
> 
> >Howdy Tom,
> >
> >I'm Ed, a NetCDF programmer...
> >
> >I would be very interested in which architecture you built netcdf on.
> >
> >Also are you using the NC_64BIT_OFFSET in the nc_create when you try
> >to create large files?
> >
> >Thanks,
> >
> >Ed
> >
> >
> >Russ Rew <address@hidden> writes:
> >
> >
> >>>To: address@hidden
> >>>From: "Tom Black" <address@hidden>
> >>>Subject: Very large netCDF files
> >>>Organization: NOAA/NCEP
> >>>Keywords: 200408171812.i7HICaaW023852 netCDF large file
> >>>
> >>Hi Tom,
> >>
> >>
> >>> Here at NCEP we are hitting major roadblock.  We need to generate
> >>> large netCDF files for very large domains as input for our
> >>> forecast models.  However in our first attempts we are stopped by
> >>> that fact that the netCDF files which are certainly over 2
> >>> gigabytes long are not created properly.  I tried installing your
> >>> beta version of netCDF3.6.0 since it was our understanding that
> >>> the 2Gb limit was no longer an issue with that version.  Although
> >>> the installation appeared to go okay, the same problem remains.
> >>> When I back off to a smaller integration domain for which smaller
> >>> netCDF files are produced, then there is no problem either with
> >>> 3.5.0 or 3.6.0.  Can you help us with information that will permit
> >>> us to write netCDF files that are well over 2Gb in length?  Thank
> >>> you very much.
> >>>
> >>Yes, We are *very* interested in the problems you are encountering
> >>with the 3.6.0 beta release.  We have been generating test cases that
> >>work as intended, but we would like to make sure the library works for
> >>realistic cases that may uncover bugs not revealed by our current
> >>tests.
> >>
> >>If you could make available an example of code that demonstrates a
> >>problem with accessing large netCDF files that we can reproduce and
> >>debug, we would be very grateful!
> >>
> >>--Russ
> >>
> >
> >
> >
> >
> 
> 
> -- 
> Thomas Black
> Meteorologist
> NOAA/NWS/NCEP/EMC
> Mesoscale Modeling Branch
> address@hidden