[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: netcdf4, java and standards





On 11/25/2013 1:53 PM, Roy Mendelssohn - NOAA Federal wrote:
Yes they are not understanding the CF DSG convention, but the point
of a possible alternate way to do it is valid.  The problem from our
point of view is just that, there a lot of alternate ways to do it in
netcdf4, and if we are going to write we would rather it to be the
agreed upon way  (right after we agree on 'groups' - sorry couldn't
resist)

yep


And writing to the netcdf4 C library means writing to hdf5 library
which means the user  (the user in this case being someone who is
installing an ERDDAP instance) has to install all of that, and down
the road you can start having issue with library versions and
locations, and we would have to figure out what is going on.  While
right now, it is all in the war file.

yep



But for now, the CF DSG defined in netcdf4 would be a start, even if
the CF people don't agree to it (but perhaps the NASA folks will).  I
have purposefully tried to keep this discussion to a small group,
because I saw the previous  discussions having to do with CF and
anything to do with netcdf4, and I can do without the flames, thank
you very much.  I would add that one way to take the comments I set
along from the user is that they see a way that exists in netcdf4
that would make the representation of the data more useful to them.
Now you can't just change standards for every person who has an idea
of what would be more useful, but the same time usage (as with
groups) may get way ahead of the standard, which would make the
standard more and more irrelevant.

no way around it i think.
CF is not going to figure this out.
we can only do it, then submit as possible standard.

ill just say that the CDM handles lots of formats and conventions that arent CF.

look at the bright side, you could call it "Roy's Conventions" and then reap the glory!



-Roy



On Nov 25, 2013, at 12:24 PM, John Caron <address@hidden>
wrote:

Hi Roy:


On 11/25/2013 9:53 AM, Roy Mendelssohn - NOAA Federal wrote:
Hi All:

For some background, here are snippets  from a user who is
accessing the CalCOFI data  (stations that are sampled by ship
for a variety of parameters) through ERDDAP, and getting back
netcdf files that follow the CF DSG specs.


I have a question for you. I am playing with the SIO
hydrographic bottle data, and did a test extract to examine
the variables in a downloaded netcdf file (both nc and ncCF
formats).

What is puzzling me is that the dimensionality of the
variable for measured oxygen, for example, is 1 rather than 4
(i.e. ) time, lat, long, depth.

ncdump shows three coordinates, as specified in the header,
implying that the dimension is 3, not 1:

float o2ml_l(row) ; o2ml_l:actual_range = 0.f, 9.44f ;
o2ml_l:coordinates = "time latitude longitude" ;
o2ml_l:ioos_category = "Dissolved O2" ; o2ml_l:long_name =
"Measured Oxygen" ; o2ml_l:units = "ml/L of seawater" ;

not sure that i understand the problem. Is s/he just asking to
understand the DSG convention?

If so, its the usual confusion about dimensions vs coordinates.


and

NetCDF4 allows multiple unlimited dimensions (which in this
case would be time and depth coordinates). The netCDF4 data are
then read as a ragged array and gridding can be done easily
because the dimensionality of the variables is known.
Essentially the data become 4-dimensional semi-random point
data rather than a regular array, and the plotting functions in
NCL or other program can handle this.


There are several things that come up in relation to this, that
I thought you folks might provide some guidance:

1.  We have been reluctant to provide hdf5 writes in ERDDAP
because there is no pure Java way to do so.  Many of the Java/C
interfaces only work so-so, but even more the chance for error
and support nightmares in making sure we can connect with the
appropriate libraries is not something we really want to do.  Is
there anything on the horizon for a pure Java write interface  -
we do not need everything  hdf5 can do, basically we need to be
able to write netcdf4 files.

CDM supports netcdf-4 writing using the netcdf C library:

http://www.unidata.ucar.edu/software/thredds/v4.4/netcdf-java/reference/netcdf4Clibrary.html



I think on server side its not so bad, and Unidata is willing to support questions and problems using it. However, you are right that CDM is not yet supporting full access to netcdf-4 features. Specific feature requests would help prioritize development there.

IMO there will not be a pure java netcdf-4 writing library, at
least not from Unidata and I doubt anyone else will be willing to
spend the significant resources on it; THG has not and they would
be the only other obvious group.



2.  hdf5 (even netcdf4) is a wide open format unless there are
some standards on how things are to be written.  The best  that
we can tell, the CF DSG  are only defined in terms of netcdf3
(are we are incorrect on this?), and some of you had the rude
awakening to what happens when you propose to try to do anything
that keeps up with some changing technology.  So even if there
aren't formally approved standards for the type of write
mentioned above, are there at least proposed and public available
"standards" that we can both use and point users to?

correct, CF is netcdf-3 only and its hard to get CF interested in
netcdf4. My guess is that "we" will just do it, and later it will
become part of CF. In a way thats better anyway, standards should
arguably endorse proven technology not invent new ones.

I would like to work on trying netcdf-4 to store point obs better,
mulitple unlimited dimensions being one such possibility. If you
want to experiment with that in ERDAP, I will do my best to
collaborate. As usual, time is limited. We have 2 new hires coming
on board in the next month and Im hoping that will give more time.

Regards, John


Thanks

-Roy


********************** "The contents of this message do not
reflect any position of the U.S. Government or NOAA."
********************** Roy Mendelssohn Supervisory Operations
Research Analyst NOAA/NMFS Environmental Research Division
Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific
Grove, CA 93950-2097

e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029 fax: (831)-648-8440 www:
http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From
those who have been given much, much will be expected" "the arc
of the moral universe is long, but it bends toward justice" -MLK
Jr.


********************** "The contents of this message do not reflect
any position of the U.S. Government or NOAA." **********************
Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS
Environmental Research Division Southwest Fisheries Science Center
1352 Lighthouse Avenue Pacific Grove, CA 93950-2097

e-mail: address@hidden (Note new e-mail address) voice:
(831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill." "From those
who have been given much, much will be expected" "the arc of the
moral universe is long, but it bends toward justice" -MLK Jr.