[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #TLB-677315]: FW: Net CDF-4 Issues

Hi Coy,

Sorry to take so long to answer these.

> Problem 2) Making TimeSeriesPoints unlimited causes the system to crash.

I was unable to duplicate this problem with the latest snapshot release, so
either it is platform specific or has already been fixed.

Since you didn't provide program source, I just took ran the
NetCDF_3_Time.nc file you provided through ncdump to generate a CDL file,
then ran that through "ncgen -c" to generate C source to create the file.

That ran OK and generated a netCDF-3 file of size as shown:

  rw-r--r--   1 russ     ustaff     24076 Apr 16 10:57 n3time.nc

Then I changed the line in the source

   size_t TimeSeriesPoints_len = 12288;

to
   /*    size_t TimeSeriesPoints_len = 12288; */
   size_t TimeSeriesPoints_len = NC_UNLIMITED;

to make it one of two unlimited dimensions (the other is "Time")
and also changed the nc_create call from

   stat = nc_create("n3time.nc", NC_CLOBBER, &ncid);
to
   stat = nc_create("n4time.nc", NC_CLOBBER|NC_NETCDF4, &ncid);

to create a netCDF4 file with two unlimited dimensions.  This compiled
fine and ran with no errors, producing a netCDF-4 file of size

   $ ./n4time; ls -l n4time.nc
   -rw-r--r--   1 russ     ustaff     90812 Apr 17 10:50 n4time.nc

The file is so much larger because of the way unlimited dimensions work.
Every variable that uses unlimited dimensions uses chunked storage, and
writing a single new value along an unlimited dimension may allocates a
new chunk containing that value.  Subsequent values use the same chunk if
in the appropriate index range.  This again points out that you may want
to use variable length types instead of unlimited dimensions to implement
your data structure.

This also explains your next problem:

> Problem 3) Making Time, Mode, Heights and SpectraPoints unlimited works
> but causes the file to grow over 10 times bigger and makes reading it
> back much, much slower. I have attached the files for you to compare. If
> you do an ncdump on them, you will see they contain identical
> information other then a few timestamps.

To help understand the use of multiple unlimited dimensions, consider a
10x10x10 variable, with each of the 3 dimensions unlimited, and assume the
chunk size has been set to 10x10x10 (default chunk sizes changed with the
release of 4.0.1, but ignore that for now).  If you extend the variable in
the first dimension, it is now 11x10x10, but the space in the file is now
20x10x10, because an extra 10x10x10 chunk has been allocated for the data.
If you continue to extend the first dimension until the variable is size
20x10x10, no more space will have been allocated, and only two chunks will
have been used.

However, if you extend the data along each unlimited dimension instead so
that your variable is 11x11x11, the space allocated in whole chunks will
now require 8 10x10x10 chunks, as if the variable was 20x20x20, so 8 times
as much storage will be required.

--Russ








Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: TLB-677315
Department: Support netCDF
Priority: Normal
Status: Closed