[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20050218: restoring headers in large netcdf files

>To: address@hidden
>cc: Bjarne B?chmann <address@hidden>,
>cc: Peter Gylling J?rgensen <address@hidden>
>From: Karsten Bolding <address@hidden>
>Subject: restoring headers in large netcdf files
>Organization: Bolding & Burchard Hydrodynamics
>Keywords: 200502190949.j1J9ncv2029855 netCDF large file header restore

Hi Karsten,

> I only got an empty mail from you.

Sorry, I'm not sure how that happened.

> I'm running a parallel ocean model producing very large files and I
> think I've run into the problem described in the mail yesterday (or maybe
> the day before).
> 
> I use the F77 interface and having taken no special measures to support
> large files. The libary version for creating the files was:
> /opt/netcdf-3.6.0

If you've taken no special measures to support large files, then the
problem may not apply to you.  In particular, the bug could only occur
if you created the files in the new 64-bit offset version of netCDF
(the "CDF2" version) by using the NF_64BIT_OFFSET flag when you call
nf_create(), as in:

  iret = nf_create('foo.nc', ,NF_64BIT_OFFSET, ncid)

or

  iret = nf_create('foo.nc', OR(NF_NOCLOBBER,NF_64BIT_OFFSET), ncid)

If you didn't use "NF_64BIT_OFFSET" in creating the files, then they
are in the "classic" netcdf format (CDF1).  It is possible to have
CDF1 files that are larger than 2 GiB.  From the netCDF FAQ:

  Was it possible to create netCDF files larger than 2 GiBytes before
  version 3.6?

  Yes, but there are significant restrictions on the structure of large
  netCDF files that result from the 32-bit relative offsets that are
  part of the classic netCDF format. For details, see NetCDF Classic
  Format Limitations in the User's Guide:
     
http://www.unidata.ucar.edu/packages/netcdf/netcdf/NetCDF-Classic-Format-Limitations.html

Also to have files with bad headers, I think you would have had to
close the CDF2 file and then later reopen it with nf_open() and either
enter define mode with nf_redef() or change an attribute value with
nf_put_att_TYPE().  If you didn't do all of these things, then I don't
think the bug I was referring to could affect you.  

However, it's possible you have identified a different bug, either in
the netCDF library or in NCO.  I'm very interested in helping diagnose
the problem.

> Here is a listing of the files:
> phy-43:~# ls -lh /nodes/{01,02,03,04}/kbk/dk_03nm/2003/01/*3d*
> -rw-r--r--  1 kbk kbk 8.5G Feb 19 00:15 
> /nodes/01/kbk/dk_03nm/2003/01/dk_03nm.3d.000.nc
> -rw-r--r--  1 kbk kbk 8.5G Feb 19 00:15 
> /nodes/01/kbk/dk_03nm/2003/01/dk_03nm.3d.003.nc
> -rw-r--r--  1 kbk kbk 8.5G Feb 19 00:15 
> /nodes/02/kbk/dk_03nm/2003/01/dk_03nm.3d.001.nc
> -rw-r--r--  1 kbk kbk 8.5G Feb 19 00:15 
> /nodes/03/kbk/dk_03nm/2003/01/dk_03nm.3d.002.nc
> -rw-r--r--  1 kbk kbk 8.5G Feb 19 00:15 
> /nodes/04/kbk/dk_03nm/2003/01/dk_03nm.3d.004.nc
> -rw-r--r--  1 kbk kbk 8.5G Feb 19 00:15 
> /nodes/04/kbk/dk_03nm/2003/01/dk_03nm.3d.005.nc
> 
> as you can see all files are bigger than 2 GB.

I would be interested in the output of "ncdump -h" for one of these
files.  Could you send me what you get when you run

  $ ncdump -h dk_03nm.3d.000.nc

for example?  I want to determine if they are properly structured CDF1
files that meet the constraints for > 2 GiB files without using the
64-bit offsets in the CDF2 format.

> Now I wanted to extract subsets of data using the nco utilities but I
> get the following error:
> kbk@sill01:/data/kbk/dk_03nm/2003/01$ ncks -v elev dk_03nm.3d.000.nc
> aa.nc
> ERROR: program exiting through nco_err_exit() which will now call
> exit(EXIT_FAILURE)
> nco_err_exit(): ERROR nco_open()
> File too large
> 
> and sure enough if I look at the header of the file I get:
> kbk@sill01:/data/kbk/dk_03nm/2003/01$ od -An -c -N4 dk_03nm.3d.000.nc 
>    C   D   F 001

It's possible you got this error because you didn't build NCO in a way
that can handle large files.  If you didn't use the right compiler
options in building NCO, you would expect to get "file too large"
errors when reading netCDF files > 2 GiB.  I don't know much about
NCO, but it should support large files if it's built to do that.  

`Have you used ncks on any other large files successfully?  You could
use the "ncgen" utility (part of netCDF) to generate a large file in
CDF1 format and then try ncks out on that.  For example, invoking
"ncgen -o test.nc test.cdl" on the following stored in "test.cdl"
should produce a large netCDF file that ncks ought to be able to read,
if built with large file support:

netcdf test {
// Last fixed variable > 2 GiB.  Expect to fail when using library
// not compiled with Large File Support.
dimensions:
        x = 1024;
        y = 1024; 
        z = 1025;
variables:
        float x(x);
        float y(y);
        float z(z);
        short var(x, y, z);
data:
  var = 1, 2, 3;
}

> You write on the web that it possible to restore the header - how do I
> do that? I'm pretty sure the files are intact since I can do a:
> ncdump -v elev dk_03nm.3d.000.nc
> and get lots of data written to the screen.

I want to make very sure the headers are bad before trying to restore
them.  ncdump should fail to correctly read a file with a bad header
once it gets to a variable with a chopped offset.  It sounds to me
like you may be seeing a different problem.  If it turns out that this
is the CDF2->CDF1 header conversion bug I mentioned in email, then I
think I can provide a program that copies the files and fixes the
headers.

> PS: I've now upgraded the netcdf library to:
> /opt/netcdf-3.6.1-beta3

Great!  I'm sorry you had to go to that trouble, but so far you're the
first person to report possible suspect files, so maybe the bug didn't
do too much harm.

--Russ