[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Paper on OpenDAP



Hi Tennessee,

On Aug 16, 2005, at 2:54 AM, Tennessee Leeuwenburg wrote:

Peter,

One further question re your 2003 paper. One of the interoperability layers you identify as "Semantic", and the comment is "more consistent" -- by this do you mean such things as conventions within formats, such as COARDS within NetCDF?

Sort of, the structural part of COARDS. I actually identify the layer that you are referring to as "Semantic-Structure". The idea here is best understood in terms of the example presented in the paper. Consider a set of 2 dimensional satellite-derived sea surface temperature (SST) fields, each field consisting of SST on a grid in, say the western North Atlantic, at one instant in time. All grids in the data set are in the same geographic projection. The data set might have all of the fields, say one per day, for a 10 years. These data could be stored a number of different ways:

1) 3650 files, each file representing one 2-d field,

2) One 3-d (lat, lon, time) file with the time dimension having 3650 elements,

3) Ten 3-d (lat, lon, time) files with the time dimension of each having 365 elements,

4) One 3-d (time, lat, lon) file, again with time having 3650 elements,...

The "Structure" layer says that all such data sets should be represented as one 3-d object. The "Semanti-Structure" layer says that they should all be represented as one 4-d object with lat always being the first dimension, lon always being the second dimension, depth the third and time the fourth. Since this is no depth in this example, the 3 layer would just have one element. The point here is that the structure is constrained by the semantics. As you move from one layer to the next, less metadata are required. The "Structure" layer does not specify the order of the variables, so metadata telling the client what the order is is required. The Semantic-Structure layer does not require such metadata. Higher up in the level structure is the metadata that describes the data, the metadata that I refer to as semantic and syntactic metadata. I don't think that it is worth making a big deal out of the Semantic-Structure level, I simply included it to make the overall picture clear. The amount of metadata required to specify the semantics of the structure is small. Furthermore, there is not a lot of agreement on the semantics of the structure which means that data are stored in a variety of semantic structures. Converting the data on the fly to a consistent semantic structure is a lot more work than converting them to a consistent structure, the next level down. I think that that is valuable and that's what the Aggregation Server does.

Do any such conventions exist for OpenDAP?

No. The idea is to let different disciplines impose whatever structure they want on top of OPeNDAP. The more appropriate question would be "Does the oceanographic (or the Earth Science) community have any such convention?" Unfortunately, the answer is again no. That is my fault. I should have recommended one a long time ago. I think that COARDS (or CF) would be an excellent starting point and COARDS is becoming a de facto standard in that many data providers use it. I was reluctant to impose COARDS on the oceanographic use of the system since that would mean the reordering of some archives which could be very costly. I prefer a modified C COARDS, one that does not adhere to the semantic structure, but does adhere to the rest of the standard.

Here at BOM we more or less have our conventions agreed on for my particular project, but if there were any recommended best practise it could be interesting.

What do you think of adopting COARDS or CF? Would one or the other address all of your data sets? If not, what is missing? If it could, but your would rather not adopt it, it would really be useful to know why and to know what you guys have adopted and why. I am hoping that the Marine Metadata effort comes up with a recommendation for the oceanographic community. I'm not sure what is being done for the meteorological community, but it would be great if there was a similar effort (to the Marine Metadata program) and if they came up with a recommendation.

Peter


Cheers,
-Tennessee

--
 Peter Cornillon
   Graduate School of Oceanography     -  Telephone: (401) 874-6283
University of Rhode Island - Fax: (401) 874-6728 Narragansett, RI 02882 - E-mail: address@hidden