[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 19990518: opening a 2.6 GB file using netCDF utilities

Nelson,

> From: "Nelson Takashi Yunaka" <address@hidden>
> Organization: Instituto Oceanografico - USP
> Keywords: 199905172301.RAA21343

In the above message, you wrote:

> I am sorry to mail to support before, visita a list
> (I read rapidly the Faq.)

I found the answer to your question in our support email archive at
<http://www.unidata.ucar.edu/packages/netcdf/index.html>.  I search for
the string "large;file".

> I would like to know if it is possible to open a 
> file aprox. 2.6 Gb with ncutilities that comes 
> with netcdf package, with solaris 2.6.
> 
> The code use automatically all functions of
> opening, writing, closing, editing files larger
> than 2Gb, automatically ?

The short answer is yes -- but file portability is compromised.  I've
enclosed a more definitive reply on this topic by the developer of the
netCDF package.

> If not you know any program that could open
> this kind of file (cdf format) and split in two
> smaller files ? (choosing variables, ranges, etc).

There are many utilities for manipulating netCDF datasets.  They are
described at <http://www.unidata.ucar.edu/packages/netcdf/software.html>.

--------
Steve Emmerson   <http://www.unidata.ucar.edu>

--------Begin reply
From address@hidden  Thu Jan 21 13:13:22 1999
        by unidata.ucar.edu (8.8.8/8.8.8) with ESMTP id NAA19909
        for <address@hidden>; Thu, 21 Jan 1999 13:13:21 -0700 (MST)
        by binnie.unidata.ucar.edu (8.8.5/8.8.5) id NAA120911;
        Thu, 21 Jan 1999 13:13:19 -0700 (MST)
From: "Glenn P. Davis" <address@hidden>
Message-Id: <address@hidden>
Date: Thu, 21 Jan 1999 13:13:19 -0700
In-Reply-To: "R. K. Owen" <address@hidden>
        "2GB limit" (Jan 21,  8:25am)
References: <address@hidden>
X-Mailer: Z-Mail (5.0.0 30July97)
To: "R. K. Owen" <address@hidden>
Subject: Re: 2GB limit and netcdf
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii



> I have a quick user question:
> Are the netCDF datasets limited by the 2GB limit?
> I've looked at the code and the lseek()s, etc. use "long"s which
> are 64 bit (on Crays) and can handle >2GB;
> however, the meta-data in the header
> is limited to 4 bytes ... which would probably limit the dataset size
> to 2GB (or 4GB if negative values are not needed for flagging exceptional
> conditions).
> Anyways, I just wanted to pass this by you before I conclude to the
> user that: no, datasets must be < 2GB.

R.K.:

As you observe, the _calculation_ of the offset of a particular datum,
and the number handed to lseek() (or ffio_seek()) is done in the system off_t.
This is 64 bits on crays and several other systems (including SGI).

The largest offset stored on disk is the offset of the
*first* record variable. This is typically well within the 2^31-1 limit.
So, it is possible to create valid files which have data beyond the 2^31-1
limit.

Warning! If you try to read such a file on a system which
has a 32 bit off_t, you are out of luck. Of course, you couldn't
have copied the file to that system without truncation anyway.

An interesting sub-case is systems like SunOS 5 and AIX, which now
support large files in the filesystem but have off_t a 32 bit quantity.
These systems include a second set of interfaces and types ( lseek64(), ...
off64_t, ...) to support the large files. Typically, these systems include
some compilation flags which map the old interfaces to the 64 interfaces.
If that is the case and the netcdf is configured and compiled appropriately,
things will work.

Warning #2! Netcdf library versions prior to netcdf-3.4 may not deal with
the offset calculation properly.

The upshot is that you can create large netcdf files,
but the file portability is compromised.

-glenn