[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 2 GB netCDF limit address@hidden

> Hi again Russ-
> 
> > > On another topic...many people here are starting to *panic* about the 2
> > > GB file limit in netCDF.  I know you are planning to remove that limit
> > > with netCDF 4, but how far off is that?  After convincing everyone here
> > > to convert to netCDF, I am naturally feeling a bit of panic, too :-) !
> > 
> > Permitting bigger files requires a file format change (64-bit offsets
> > instead of 32-bit offsets).  We don't want to have to change the netCDF
> > file format more than once, if we can help it.  So the file changes
> > required for netCDF 4 will anticipate everything we can think of.  It
> > may be more than a year away.  I'm afraid I can't give you anything more
> > definite now ...
> 
> Well, we've spent quite an interesting afternoon here trying to figure
> out just what happened to a user's "extra-large" netCDF file.
> Basically, he concatenated 2 files, each ~1.2GB, on our Cray T90 using
> "nccatm", a local extension (for multiple files) of Chuck Denham's
> "nccat" program.  Panic initially set in when the resulting file did
> not appear readable, based on attempts to ncdump it from our SGI
> workstations.  However, we soon found that we *could* ncdump it on the
> Cray!  Encouraged, we split the 2.4GB file into two parts using
> Charlie Zender's "ncrcat" utility, and both pieces were readable from
> our SGI's (we viewed them using "ncview").
> 
> Not believing our good fortune, we repeated this user's steps with some
> new files, and checked the status code returned by "nccatm" : 0!  With
> the wind apparently at our backs, we further checked the return code
> coming back from ncvarput while it was writing well beyond 2GB : 0!
> 
> We are continuing to concatenate as I write this (4.5GB and counting).
> 
> Can we infer from this that files larger than 2GB *are* possible on a
> Cray?  (Obviously, we'd like to find out that our seeming success is by
> design, not by accident.)  Despite an inability to access such files
> from our workstation, this would be very good news :-)
> 
> John

The on disk-storage of offsets in netcdf Version 1 file format is
a 32 bit quantity. This should be limited in software to the signed
positive range 2^31-1 as it is written to disk. netcdf-2 may
not have been careful about this, and there may be a netcdf bug
on the cray which allows numbers up to 2^32 to be written out.

However, I don't think this is what is going on.

The _calculation_ of the offset of a particular datum, and the number
handed to lseek() (or ffio_seek()) is done in the system off_t.
This is 64 bits on crays and several other systems (including SGI).
The largest offset stored on disk is the offset
of the *first* record variable. This is typically well within the 2^31-1
limit. So, it appears to be possible to create valid files which have data
beyond the 2^31-1 limit. If you try to read such a file on a system which
has a 32 bit off_t, you are out of luck. (But, if that is the case, the
file should have been truncated on copy to the system.)

So, you are wondering why it did not work on the SGI?
I am too. On modern SGI's, the off_t is 64 bits irrespective of
the -64 or -32 compiler flags. If you were using the netcdf-3 library
to read the big file, it should work. (For netcdf-2, we did the offset
caluclation in a 'long', so it probably wouldn't work.)

-glenn