[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: WP21B: Conventions tag in OPERA datamodel

Hi all:

This is a bit complicated for casual perusal. When you get a sample file, it 
will be much easier for
me to analyze it and give feedback.

In the meanwhile, id like to comment on "Endnote on software compatibility and 
versions" section:

"Since we recognize the benefits of being able to manage HDF5 files with NetCDF 
(and vice versa), we
require that any implementation of the information model specified in this 
document must use HDF5
version 1.8.0 or later" etc.

Unfortunately, as I mentioned before, the HDF5 information model does not 
include shared dimensions,
and there are some hoops to jump through to compensate. HDF-EOS, for example, 
has proprietary
metadata in order to add equivalent semantics (i wouldnt recommend their 
approach, but it
illustrates the fact that some approach is probably needed). I am unable to 
tell from a casual
reading of the doc what you are doing about that, which is why a sample file 
will clarify things for
me.

You might also consider using the netcdf-4 library as your API, which would 
solve this problem, as
it adds shared dimensions within the HDF5 format. You also might want to check 
in with the HDF5
group about this issue.

Regards,
John


Daniel Michelson wrote:
> Hej,
> 
> I can only agree with the previous speakers; this is a great step
> forward! Attached is the latest draft. I only just realized that I
> haven't distributed this latest version after making the changes a few
> weeks ago. The only change from the version of Dec 21 is adding T to
> Table 15...
> 
> Hans and Ernst, please take the ball and run with it, ensuring that our
> use information model will be manageable by NetCDF too.
> 
> If you need my LaTeX source to introduce improvements to the document,
> please let me know.
> 
> best, >-D
> 
> Holleman, Iwan (KNMI) wrote:
>> Hi Daniel, Hans, Maciej,
>>
>> Cc John, Ernst,
>>
>> I am delighted to read this response from John Caron as it appears
>> pretty straightforward to make the HDF5 information model netCDF
>> compatible. When we have done so we should start advertising the
>> information model as a universal “HDF5/netCDF” model for weather radar
>> (change the documents and titles etc) . I am sure that this will give
>> us additional momentum during some of the discussions that we and/or I
>> have to face.
>>
>> Best regards,
>>
>> Iwan Holleman
>>
>> Program Manager OPERA
>>
>> -----------------------------------------------------------------------
>>
>> dr. I. Holleman
>>
>> KNMI, Netherlands
>>
>> P.O. Box 201, NL-3730 AE, De Bilt
>>
>> T +31 30 2206818  
>> W ___www.knmi.nl/~holleman_ <http://www.knmi.nl/~holleman> en
>> ___www.knmi.nl/opera_ <http://www.knmi.nl/opera>
>>
>> -----------------------------------------------------------------------
>>
>> _____________________________________________
>> ***From:* Beekhuis, Hans (KNMI)
>> ***Sent:* maandag 16 februari 2009 16:38
>> ***To:* Daniel Michelson; 'Maciej Szewczykowski'
>> ***Cc:* Holleman, Iwan (KNMI); John Caron (address@hidden);
>> Vreede de, Ernst (KNMI)
>> ***Subject:* WP21B: Conventions tag in OPERA datamodel
>>
>> Hello Daniel, Maciej
>>
>> I asked a Ernst de Vreede, a colleague of mine to review the OPERA
>> weather radar information model , especially with a view how to have
>> the data model
>>
>> accessible by NETCDF (library's).
>>
>> Ernst forwarded  this question to his contact at Unidata and got as a
>> reply that the minimum we should do is to add in the /root of our
>> model a string attribute "Conventions"
>>
>> This attribute should be really in the /Root so any library accessing
>> this file can decide how to handle. As Unidata is maintaining Java
>> based library's for (de)coding NETCDF it is a big advantage when the
>> OPERA model can be recognised by NTCDF readers. So I think we should
>> add this tag.
>>
>> The email exchange between John and Ernst is included below.
>>
>> Don’t blush Ernst is impressed with the data model description as it
>> is now
>>
>> May be its wise to ask if Caron can have an reviewing eye on the draft
>> document for the data model?!
>>
>> --------------------------------
>>
>> Sent: tuesday 10 February 2009 18:30
>>
>> to: Vreede de, Ernst (KNMI)
>>
>> CC: Unidata netCDF Java Support
>>
>> subject: Re: European radar data standard
>>
>> *Hi Ernst:*
>>
>> *Funny you should mention OPERA and BUFR. Someone just asked me to
>> read some OPERA/BUFR files (apparently radar?) but i couldnt because
>> they are putting a missing value into the center id, and using
>> non-standard descriptors that require you (me) to use special tables
>> based on the ... you guessed it, the center id. Anyway, Ive written
>> someone there (havent heard back) but if you have any friends who can
>> help fix that problem, i would appreciate it.*
>>
>> *Anyway, thats a side topic, some comments are embedded:*
>>
>> Vreede de, Ernst (KNMI) wrote:
>>
>>>  Hi John,
>>
>>>
>>
>>>  A European project (OPERA) is busy harmonizing all European radar 
>> products into one (or two, some parties insist on BUFR!) format. A few
>> colleagues of mine are participating in te Opera project.
>>
>>>  One of them asked me to take a look at the draft version for the data 
>> format.
>>
>>>
>>
>>>  To my surprise a it is simple and readable document, describing the 
>> envisioned dataformat. Because HDF5 is already used for radar at KNMI
>> and in the Baltic states it is an HDF5 format. That's good. A simple
>> hierarchy has been devised, which appears to be very complete.
>>
>> *JC> its nice when you see competent work.*
>>
>>>
>>
>>>  After hearing about NetCDF4 and especially netcdf-java 4.0 at the 
>> workshop in november, I reckoned it might be worth to try to add a
>> minimal set of tags to the HDF5 file to make the files easier to
>> recognize for the netcdf-java library. As far as I know there is
>> currently no NetCDF convention for radar.
>>
>>>  I have some experience with the radar data structures in CDM; what 
>> I've seen seems to be mappable between the Opera format and CDM
>> without too much work.
>>
>> *JC> The big problem with HDF5 is they dont have shared dimensions
>> like netcdf. For simple cases one can use dimension scales, so i hope
>> thats possible here. Otherwise we have to develop a little bit of
>> metadata to clarify this to the CDM.*
>>
>>>
>>
>>>  I am wondering if you could help me out a little here.
>>
>>>  I think that at least a global Conventions attribute is needed. The 
>> value of that attribute should be some unique name. Does Unidata keep
>> track of those names? This tag would then make it possible for the
>> software to select the right conversion module in the software (I've
>> been trying this already with our KNMI HDF5 format for radar volume
>> scans, but haven't finished that yet).
>>
>> *JC> yes we track them, very important for generic software! add an
>> attribute in the root group named "Conventions", with a String value
>> something unique like "OPERA Radar v1.0". Then write up something
>> about it, put it on the web, send us a URL to it and we put it on this
>> page:*
>>
>> * *
>> ***_http://www.unidata.ucar.edu/software/netcdf/conventions.html_*****
>>
>> *Alternatively, one could use the existing CF conventions, at least
>> for the coordinate system specification. Its quite easy, I can show
>> you how if interested.*
>>
>>>
>>
>>>  Are there any other tags that are really needed?
>>
>> *JC> have a look at this document:*
>>
>> * *
>> ***_http://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html_*****
>>
>>
>> *the main things from my POV are clear specification of the
>> coordinates, and units on the variables, preferable udunits compatible.*
>>
>>
>>>
>>
>>>  I am not in a position to change the whole format, because a lot of 
>> work has already been done, but a few adaptations to make the format
>> more easy to use in netcdf-java 4.0 (and then IDV and all those other
>> nice applications) should be possible.
>>
>> *JC> With some small changes, we can probably get this to work. can
>> you send me a sample file?*
>>
>>>  [ Of course any netcdf-java implementations for the new format will be
>>
>>>  sent to Unidata!!]
>>
>> *JC> that sounds great.*
>>
>>>
>>
>>>  Thanks in advance, Ernst de Vreede
>>
>>>
>>
>>>
>>
>> --------------------
>>