Showing entries tagged [netcdf]

netCDF Identifiers and Character Escape Mechanisms (sigh!)

netCDF Identifiers and Character Escape Mechanisms (sigh!)

Ideally, netCDF should allow any printable UTF-8 character to be used in an identifier. Currently, that is almost the case, with forward slash being the exception because of the syntax of HDF5 identifiers.

More and more, the netCDF API is being used as wrapper for a wide variety of other formats: HD5, HDF4, GRIB, BUFR, DAP2, DAP4, etc. During the process of defining translations to/from netCDF and these other format, it is necessary to implicitly or explicitly define netCDF identifiers from the schemas of these other formats.

The canonical example is HDF5. In HDF5, many API functions take a path, which is a sequence of identifiers separated by '/'. A path may be absolute ("/g1/g2/x") or relative ("y"). It appears to be the case that there is no way in HDF5 to specify an identifier containing '/', such cases are always interpreted as paths. So, if one naively defined, thru the netcdf-4 API, a variable named "/x/y", there is no apparent way to actually get this defined properly in HDF5. It is this fact that has led to the current, IMO undesirable, restriction that netCDF identifiers may not contain '/'.

Super Escapes

This situation is going to recur as the netcdf API is used to wrap other data formats. What we will need is a mechanism by which we can convert an identifer containing arbitrary UTF-8 characters into another identifier in some rather restricted set of legal identifier characters. In addition, I would impose the rule that the conversion is invertible.

This kind of "super-escaping" is very hard because in the worst case, we are likely to encounter the situation where legal identifier characters are restricted to something like the alphanumerics plus underscore.

Indexes R'nt us

Indexes vs. indexes.  Gray's third requirement:

3. Intelligent indexes: for efficiently subsetting and filtering the data.
[Read More]

NetCDF CDL vs SQL

Gray's second desiderata gloria scientia: 

2. Schema language: powerful data definition tools allow one to specify the abstract data formats and to specify how the data is organized.

[Read More]

Whats wrong with Fortran unformatted writes?

Talking about the first of Gray's criteria for scalable Scientific Data Management Systems:

1. Data independence: the physical data organization can change without breaking existing applications

 

[Read More]

Great YouTube of ToolsUI use!

Here's a great demo, less than 5 minutes long, of Tools UI and using NcML: http://www.youtube.com/watch?v=w-Df4Np7tLA

 

Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« May 2024
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
 
       
Today