[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [netcdfgroup] Netcdf 4 examples

Thanks Russ! The first link in particular is the type of information we are looking for. There are now quite a few ways to represent he same types of data, but it would be nice if there were consistent ways for doing so, plus some of the ways will be more efficient in terms of access, compression, ability to use chunking etc. etc.
Also, based on the python interface that Jeff Whitaker wrote and his test programs (while the calls are in python the bulk of the dirty work is done by the C libs), read access to a file using compression on a large array was roughly 8-10 times slower than the same array uncompressed, and the savings in size was roughly one order of magnitude. Does this sound like what you would expect? The compression feature is very attractive given the size of our files, but that is a reasonably large speed hit.
Now to use some of these new features, I am either going to have to learn to program in C for real or get the wrappers I am working on to work. 

Thanks again,

-Roy

On Dec 27, 2007, at 10:00 AM, Russ Rew wrote:


Roy,


The new features of netcdf-4, such as user-defined types, compound
types and variable length arrays,  open up a lot of possibilities for
storing data, such as profile data that is taken at a variable number
of depths.  But as is often the case,  when there are a lot of
possibilities, usually some uses of these features make for better or
more efficient design then others.  We are looking for example files
and/or code that use some of these new features in netcdf-4 to see
different approaches that people have taken.

Any help greatly appreciated.
I was away from my email when you posted this question, but I have a few 
suggestions.

First, I have a draft of a document on "Developing Conventions for
NetCDF-4" that has some examples and discusses some issues in use of
new netCDF-4 features:
http://www.unidata.ucar.edu/software/netcdf/docs/nc4- conventions.html
Second, I'm trying to construct some more realistic examples for testing 
with the netCDF-4 version of ncdump, including these:

  tst_comp.c
  tst_vlen_data.c
  tst_string_data.c
  tst_opaque_data.c
  tst_enum_data.c
  tst_group_data.c

each of which creates a netCDF file that ncdump is tested against with
output expected as

  ref_tst_comp.cdl
  ref_tst_vlen_data.cdl
  ref_tst_string_data.cdl
  ref_tst_opaque_data.cdl
  ref_tst_enum_data.cdl
  ref_tst_group_data.cdl

These should all be available in the ncdump directory of recent
releases.

Third, a talk I gave in June on "Best Practices" contained some
suggestions for netCDF-4:

  http://www.unidata.ucar.edu/presentations/Rew/bp-seminar.pdf
I hope these are of some help. In any case, I'd appreciate feedback. I 
realize I still need some more realistic examples, e.g. for the Dapper
conventions for soundings or for some of John Caron's observational data 
conventions for netCDF-3.  We need real users trying out some of these
features and reporting back on what's actually useful.

--Russ


**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA." 
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division 
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."