[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: DODS, netCDF and Large File Support



Russ Rew wrote:


To: address@hidden, address@hidden, address@hidden, address@hidden,
      address@hidden
From: Rick Grubin <address@hidden>
Subject: Re: 20040707: DODS, netCDF and Large File Support
Organization: NCAR SCD
Keywords:200407071325.i67DPXaW025904 dods lfs linux


Hi Rick,


[All of the following refers to Linux hosts, particularly RH9 and FC1]

I have a client application (NCL http://ngwww.ucar.edu/ncl) which uses
netCDF as its base file structure. 
Earlier this year, I was able to successfully integrate DODS client
libraries into NCL, giving users access to remote files.

Recently, users have been asking for large file support ("lfs", >2Gb)
within NCL.  I've also been able to integrate this support.
In each of the two cases noted above, the feature in question (DODS and lfs) was integrated independent of the other, and each worked/works well. 

When trying to build both into my client application at the same time, I
note that DODS continues to work as expected, but access to large files
fails.

My netCDF libraries (v3.5.1) are built with lfs, as are my DODS libraries
(v3.4.3).  By "built" I mean I compile with:

   -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE

in order to 'turn on' lfs on a 32-bit host.  When I try to access a large
file, I see the following message:

   lnetcdf/lposixio.c:269: px_get: Assertion `offset >= 0' failed.

I suspect that this is a file size error, and for some reason a file size
is a negative number.  I expect I'll build the netCDF libs with debugging
and trace this through.

My questions:

1.  Are DODS and lfs mutually exclusive?
1a. Is lfs supported with DODS file access?
2.  If 'no' to #1/#1a, what (obvious?) item am I overlooking?


I think DODS and LFS should work together.  Charlie Zender's NCO
package

 http://nco.sourceforge.net/

claims to support both LFS and NCO, though the LFS support has just
been added recently.  The LFS support in netCDF 3.5.1 is somewhat
limited (only one non-record variable over 2 GB allowed, etc.).  We
are adding better LFS support to netCDF 3.6.0, which will be released
later this month. 
None of this explains the bug you are seeing, which I haven't seen
before.  The assertion in lposixio.c/px_get is from DODS code rather
than the netCDF library, since netCDF has know "lposixio.c" file.

If you discover more by tracing this, please let us know what you
find. 
--Russ


Hi all,

I'm CCing James Gallagher on this to get another OPeNDAP/DODS perspective.
The netcdf file libsrc/posixio.c is used as the OPeNDAP file nc3-dods/lnetcdf/posixio.c with very few changes. The assertion that fails above is the same in both versions of the file and none of the other changes jump out at me as being likely to affect this assertion. However, I haven't looked at any differences in other files between the netCDF code and the code as it is in OPeNDAP. 

James, any thoughts on OPeNDAP and LFS?

Ethan

--
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------