[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: large netCDF data sets

>To: address@hidden
>From: "Tom Black" <address@hidden>
>Subject: Re: 20040824: large netCDF data sets
>Organization: NOAA
>Keywords: 200408241744.i7OHiDXn015989

Tom,

> Thank you Russ.  It was largely that .inc file that I wasn't sure 
> about.  Without setting any of the environment variables, I compiled 
> with xlf (I couldn't use xlf90 due to the continuation lines).  It 
> compiled okay and ran with only one complaint: "Invalid argument".  
> However a wrf_test.nc file of length 2.147 Gb was written.  This would 
> seem to be good news.  My WRF SI job still chokes for some reason.  
> Unlike the test job, it writes out three netCDF files in succession.  
> The first one written is very coincidentally also 2.147Gb long.  The 
> second and third ones are tiny (6220 bytes).  The fact that the files 
> from wrf_test and the large file from the SI job are almost the same 
> length seems very strange.

Nope, not good news.  The "invalid argument" is a system error that is
returned from a write attempt beyond the 2 GiB limit.  The size of
wrf_test.nc should be more like 2661494168 bytes, which is what it is
when we run it.  The size of file you are seeing is really the "2GB"
limit, which more accurately is 2**31, or 2147483648.

First, are you sure your file system is configured to support large
files?  It's possible that no file larger than 2**31 bytes can be
written on the file system you are trying to write with any software,
if the file system is not capable of storing such files.

Second, how did you build the netCDF-3.6.0-beta3 library, that is what
environment variable settings did you use before invoking the
"configure script"?  Specifically, we would like to know the values of
any of these environment variables:

 CC CFLAGS CPPFLAGS FC FFLAGS CXX CXXFLAGS F90 F90FLAGS

It may be that the library you linked against wasn't built with the
right compiler and linker flags to support large files, but the values
of the above flags and maybe the output from when you ran the
configure script and "make" would help us determine that.

--Russ