Re: [netcdf-java] NcML location attribute and IOSPs

Doug Lindholm wrote:
I've been developing a few AbstractIOServiceProvider-s (with NetCDF Java 4.1) to use in my NcML for some custom datasets. I've run into a few issues (some of which I've already discussed with the developers) that I'd like to get some insight on.

I have an IOSP that reads data from our relational database. The first issue I ran into was setting the "location" attribute in my NcML. There is no file and anything that wasn't a file made netcdf unhappy. I eventually found that "/dev/null" got me around that problem.

That got me thinking about the possibilities of "location" that weren't tied to a RandomAccessFile. I'd like to think that a "location" could be any URI (or URN?) that the IOSP would know how to deal with.

I then thought that the "location" could be a file that contains the URI that the IOSP could then read. I didn't like the idea of maintaining yet another file, so I figured I could use the ncml file itself. Unfortunately, netcdf is too clever to let me use an ncml file that refers to itself. Could/should that restriction be relaxed when an "iosp" is defined?

On a related note, there are often other bits of metadata that I need for my IOSP when I am serving some data files (e.g. flat binary with no metadata). Again the ncml file itself is the most practical source of the metadata my IOSP needs to define variables and such, so I don't have to keep a redundant set of information. In this case, I am using the "location" for the data file. The only option I found, beyond a naming convention, is using the "iospParam" attribute to encode the location of the ncml so my IOSP can read it. This hack is less than ideal. I'd like to think that the NetcdfFile that the netcdf code provides me in the "open" method would have a handle on the ncml file that defines it. After all, it is an instance of NcMLReader$NcMLNetcdfFile. Do you see any problem exposing the ncml location (or even the netcdf Element)? It looks trivial to add an arg to the NcMLNetcdfFile constructor and a getter to retrieve it.

Any other thoughts or workarounds to give the IOSP more flexibility to serve datasets that can't easily be represented by single location that is mapped to a RandomAccessFile?
The problem is that a location doesnt specify the IOSP needed to handle it. We have to ask all the IOSPs we know about to find out who handles it. The IOSP needs to examine the contents of the dataset to decide if "its mine". Rather than having each IOSP open the file, we pass each the RandomAccessFile, for efficiency. The IOSP design is obviously oriented to using RandomAccessFile.

Now an NcML file can specify the IOSP, so potentially this might be a workaround this problem. The presumed use of NcML is to wrap an existing NetcdfFile. You want to reverse that and let your IOSP's NetcdfFile wrap the NcML, by giving it access to the NcML.

I could see trying an experimental version that passes the NcML to the iosp specified in the NcML. I would guess that NcMLReader would not further wrap it, though maybe we can relax that later. Im not sure what issues we'd find, and but we could try it, though i wouldnt commit to supporting it until we have some experience.

The question is whether an IOSP is the right level to implement your database access. Alternatively, one could subclass NetcdfFile, which is what other non-file implementations like opendap, thredds, and cdmremote do. The problem is that theres no mechanism to associate the location with a factory that creates those subclasses. I suppose we could create one, though.

So have you considered subclassing NetcdfFile instead of writing an IOSP ?




  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: