[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 980430: HDF vs netCDF

>To: "'address@hidden'" <address@hidden>
>From: Dan Blanks <address@hidden>
>Subject: HDF vs netCDF
>Organization: Nanoelectronics Group, Raytheon Systems
>Keywords: 199804302200.QAA01981

Hi Dan,

> HDF vs netCDF -- which is better?
> 
> I am trying to determine whether I should be using
> HDF of netCDF for a program that must store large
> amounts of scientific data.  Most of this data is
> in the form of floating point numbers and will
> occupy roughly 1 GByte per file, with the potential
> of hundreds of files before our project is done.
> These data sets are typically 2-dimensional on a
> homogeneous x-y grid, but I will be generating them
> as a function of time.  So animation capabilities are
> needed to view the progress of our simulation.
> 
> I will use a C routine to write and read these
> data files.  I would like to use a format that is
> readable by affordable plotting packages like IDL,
> Fortner's HDF tool, AVS, etc. to display these data
> sets.
> 
> My research keeps pointing me to HDF or netCDF, but
> it's unclear whether one is superior to the other or
> if it matters which I pick.  If you receive this
> message, then you are a proponent of
> one format or the other.  Can you advise me on the
> wisdom of picking one or the other?  If you need
> more information on my project, please feel free
> to ask.

I don't think either HDF or netCDF is clearer superior for all purposes.
From your brief description, it sounds as if either would be adequate
for your data.  It might be best to try writing a file of data in each
format to see which interface you like best, which interface
documentation you prefer, and to make sure that performance on the
platforms you will be using is adequate.

I think that the netCDF interface is somewhat simpler, because its data
model is simpler.  But the more complex HDF data model makes it better
for storing some kinds of data structures, for example multiple tables
that must grow and shrink independently.  You also need to distinguish
between the current HDF4 and the next generation HDF5 that uses a new
data model and a new format.

Data access with netCDF is very efficient for most common patterns of
access.  However, I/O performance is not much of an issue for many
applications.  HDF has extra code for compression and "chunking"
(accessing multidimensional data in generalized tiles so that there is
no favored direction for access).

NetCDF has C++ support, which I think HDF lacks.  There is also an
experimental 100% Java implementation of netCDF available for testing.

HDF has one big customer, NASA's EOS project, so there may be more
developers working on it and developing tools for HDF data.

I hope this helps ...

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu