[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: netcdf4, java and standards

Hi Roy:


On 11/25/2013 9:53 AM, Roy Mendelssohn - NOAA Federal wrote:

Hi All:

For some background, here are snippets  from a user who is accessing
the CalCOFI data  (stations that are sampled by ship for a variety of
parameters) through ERDDAP, and getting back netcdf files that follow
the CF DSG specs.



I have a question for you. I am playing with the SIO hydrographic
bottle data, and did a test extract to examine the variables in a
downloaded netcdf file (both nc and ncCF formats).

What is puzzling me is that the dimensionality of the variable
for measured oxygen, for example, is 1 rather than 4 (i.e. )
time, lat, long, depth.

ncdump shows three coordinates, as specified in the header,
implying that the dimension is 3, not 1:


    float o2ml_l(row) ;
        o2ml_l:actual_range = 0.f, 9.44f ;
        o2ml_l:coordinates = "time latitude longitude" ;
        o2ml_l:ioos_category = "Dissolved O2" ;
        o2ml_l:long_name = "Measured Oxygen" ;
        o2ml_l:units = "ml/L of seawater" ;
not sure that i understand the problem. Is s/he just asking to understand the DSG convention? 

If so, its the usual confusion about dimensions vs coordinates.



and


NetCDF4 allows multiple unlimited dimensions (which in this case
would be time and depth coordinates). The netCDF4 data are then
read as a ragged array and gridding can be done easily because the
dimensionality of the variables is known. Essentially the data
become 4-dimensional semi-random point data rather than a regular
array, and the plotting functions in NCL or other program can
handle this.



There are several things that come up in relation to this, that I
thought you folks might provide some guidance:

1.  We have been reluctant to provide hdf5 writes in ERDDAP because
there is no pure Java way to do so.  Many of the Java/C interfaces
only work so-so, but even more the chance for error and support
nightmares in making sure we can connect with the appropriate
libraries is not something we really want to do.  Is there anything
on the horizon for a pure Java write interface  - we do not need
everything  hdf5 can do, basically we need to be able to write
netcdf4 files.


CDM supports netcdf-4 writing using the netcdf C library:

http://www.unidata.ucar.edu/software/thredds/v4.4/netcdf-java/reference/netcdf4Clibrary.html
I think on server side its not so bad, and Unidata is willing to support questions and problems using it. However, you are right that CDM is not yet supporting full access to netcdf-4 features. Specific feature requests would help prioritize development there.
IMO there will not be a pure java netcdf-4 writing library, at least not from Unidata and I doubt anyone else will be willing to spend the significant resources on it; THG has not and they would be the only other obvious group. 




2.  hdf5 (even netcdf4) is a wide open format unless there are some
standards on how things are to be written.  The best  that we can
tell, the CF DSG  are only defined in terms of netcdf3  (are we are
incorrect on this?), and some of you had the rude awakening to what
happens when you propose to try to do anything that keeps up with
some changing technology.  So even if there aren't formally approved
standards for the type of write mentioned above, are there at least
proposed and public available "standards" that we can both use and
point users to?
correct, CF is netcdf-3 only and its hard to get CF interested in netcdf4. My guess is that "we" will just do it, and later it will become part of CF. In a way thats better anyway, standards should arguably endorse proven technology not invent new ones.
I would like to work on trying netcdf-4 to store point obs better, mulitple unlimited dimensions being one such possibility. If you want to experiment with that in ERDAP, I will do my best to collaborate. As usual, time is limited. We have 2 new hires coming on board in the next month and Im hoping that will give more time. 

Regards,
John



Thanks

-Roy


********************** "The contents of this message do not reflect
any position of the U.S. Government or NOAA." **********************
Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS
Environmental Research Division Southwest Fisheries Science Center
1352 Lighthouse Avenue Pacific Grove, CA 93950-2097

e-mail: address@hidden (Note new e-mail address) voice:
(831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those
who have been given much, much will be expected" "the arc of the
moral universe is long, but it bends toward justice" -MLK Jr.