Re: netcdf unlimited dimension in versions 3 & 4 with fortran90

  • To: Ed Hartnett <ed@xxxxxxxxxxxxxxxx>
  • Subject: Re: netcdf unlimited dimension in versions 3 & 4 with fortran90
  • From: Jeff Whitaker <jswhit@xxxxxxxxxxx>
  • Date: Wed, 01 Aug 2007 15:07:50 -0600
Ed Hartnett wrote:
Ken Knapp <Ken.Knapp@xxxxxxxx> writes:

Hi-
I'm kind of new to writing unlimited dimensions in netcdf and ran into some odd behavior when attempting to write unlimited dimension with netCDF4 and Fortran90. In short, I find two problems (for details of the problems see below examples and code): 1) writing unlimited dimensions in netCDF4 significantly increases file size over same data written with version 3.6. I realize I could turn on deflate ... but why would netcdf4-unlimited be 10x bigger than the ver.3.6 or ver.4??

The file size is much greater for tiny files, but not that big for
larger files.

The netCDF-4 format is HDF5, which has much more header information
than netCDF-3 classic format files. So a very small file (i.e. one
without much data) will be bigger in HDF5 due to the increased header
information.

But once you start writing data to the file, the importance of the
header is less, and when you are writing reasonably sized data files
there should not be too much difference.

For example, with a 600 x 1200 array of ints, I get the following
sizes:

  -rw-r--r--   1 ed ustaff  2880104 Aug  1 13:59 tst_unlims_3.nc
  -rw-r--r--   1 ed ustaff  2917551 Aug  1 13:59 tst_unlims_4.nc


2) writing large amounts of unlimited data seems impossible? Is there a performance issue? I realize my code is likely not optimized. Maybe I'm using the wrong netCDF flags? When attempting to write 1 million points with an unlimited dimension, ver3 file (not shown) completes quickly, ver.4 file never completed.


Hmmm, I am having the same problem. I will look into this some more
and get back to you.


Thanks,

Ed
Ed:

For what it's worth, I'm also seeing very poor performance writing large chunks of data along the unlimited dimension in NETCDF4 and NETCDF4_CLASSIC (using the python interface, which is just a wrapper around the C interface). Works fine if the format is NETCDF3 though.

So, I don't think it's an issue with the fortran interface.

-Jeff

--
Jeffrey S. Whitaker         Phone  : (303)497-6313
Meteorologist               FAX    : (303)497-6449
NOAA/OAR/PSD  R/PSD1        Email  : Jeffrey.S.Whitaker@xxxxxxxx
325 Broadway                Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web    : http://tinyurl.com/5telg

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================


  • 2007 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: