Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Ian, Sounds fine to me, so long as that means I can have access to the L2 (swath) data. I am quite happy to project the data myself; the essential thing is that most geophysical parameters such as Chla must be derived from the various frequencies of water leaving radiance before the data is projected; a similar concern applies to SST and cloudmasks. cheers Dave ----- Original Message ----- From: Ian Barrodale <ian@xxxxxxxxxxxxx> Date: Friday, February 9, 2007 4:11 pm Subject: Re: Meeting about improving the GRD API. > Hi All: > > Based on your feedback (John and Ted), we've been talking and have > an interesting project idea - brought up today by Peter. > > If we were able to put GRD database support into the Java netCDF > library and use THREDDS and TDS to serve data, we may > also be able to provide gridded data reprojection support. > > What do you think of this idea please? Some details follow below. > > If a grd:// URL was allowed to contain a bit of spatial reference text > so that not only did it specify the database server, login info, > databasename, and variable name, but also the projection and its > parameters(sort of like an OPeNDAP subset specification) then the > GRD could > reproject any of the gridded data (including satellite swath data such > as from a POES satellite) to a projection of the user's choice. The > spatial reference text could be tacked onto the grd:// URL in the > recordsreturned by THREDDS according to some number of pre-specified > projections that we create. > > Such a capability would help to support users like Dave Foley and > Roy Mendelssohn (copied) who have expressed an interest to us in > storingunmapped satellite data in a spatial database, as well as other > groups such as the NOAA/NESDIS CLASS archive. According to Peter, > the NOAA CoastWatch technical group has been talking for years about > how nice it would be to be able to pull unmapped satellite data out > of CLASS > in a user- specified projection rather than having to create mapped > products. > Have a great weekend. > > Regards, > Ian > ==================== > > > > > At 08:13 AM 2/9/2007, John Caron wrote: > >Hello all, comments are in-line: > > > >Ian Barrodale wrote: > >>Hi Ted, John, Russ, and John: > >>Thank you all for taking the time yesterday to both listen to our > >>story and to further enlighten us about your work. It was much > appreciated.>>The note below provides a possible implementation > route, and some > >>questions. Please feel free to point out any shortcomings in our > >>proposed approach, and please provide any answers that come to > mind > >>regarding our questions. > >>Thanks again, > >>Ian > >>====================== > >> > >>Goal > >>------- > >>Based on feedback from BCS Grid DataBlade customers and, in > >>particular, Ted Habermann, we feel that there may be some value > in > >>providing alternate ways of accessing data from a Grid DataBlade > >>(GRD) - powered database through existing widely-used protocols > and > >>methods. Note that by "accessing", we really mean just the > >>reading part, as we already provide, through the BCS Gridded Data > >>Loader client, a means of conveniently ingesting data from many > >>forms into a GRD-powered database. One method of accessing the > >>data would be to cast it in the form of the Common Data Model > >>(CDM) supported by the Java netCDF API from UCAR. The advantage > >>of this is that: > >> * users would be able to write software using the Java > netCDF API > >> (which is fairly straightforward to use and well > documented) for > >> accessing GRD data, and > >> * data providers can use a GRD-powered database and provide > access>> to it through OPeNDAP, WCS, netCDF files, etc. using > the Java > >> netCDF API (see page 53 attachment, modified from the > slide on > >> page 53 of > >> > http://www.unidata.ucar.edu/staff/caron/presentations/CDM.ppt).>>Our > understanding of a possible implementation > >>------------------------------------------------------------------ > --- > >>To handle GRD data from the Java netCDF API, we would have to: > >>(i) Create a GRD I/O service provider for the Java netCDF API > (see > >>page 38 attachment) that can communicate with the GRD database > >>using a combination of JDBC and the existing Java GRD API. The > >>Java netCDF API uses a service provider architecture to handle > >>reading multiple different file formats and casting them in the > >>form of the CDM. > >>(ii) Create a GRD content manager to handle the georeferencing > >>information in the GRD. > >>One possible method for allowing users to access GRD data without a > >>full THREDDS catalog is to supply some type of unique URL to the > database:>> grd://user:pass@server/database > >>and the service provider would construct a CDM instance that > >>contains a main group of all the grids in the database and allow > >>the user to access those grids through the API. > >>For example: > >> grd://peter:test123@xxxxxxxxxxxxxxxxxx/coastwatch > >>might be a reference to a GRD database running at Barrodale that > >>contains gridded NOAA CoastWatch satellite-derived data for some > >>number of geographic areas and time periods. The resulting > netCDF > >>dataset would be one that contains a list of grids under a root > >>group like a directory structure: > >> / > >> /sst/ > >> /sst/northeast/ > >> /sst/northeast/jan01_2007 <---- a grid > >> /sst/northeast/jan02_2007 <---- another grid > >> ... > >> /chlorophyll/northeast/jan01_2007 <---- a third grid > >> /chlorophyll/northeast/jan02_2007 <---- and so on > >>It depends on the desired complexity of the grids in the database > >>as to whether the user would require a more sophisticated catalog > >>with querying ability such as that which THREDDS could supply. > > > >see the last answer below. > > > >BTW, the TDS will soon have the ability to do proper HTTP-based > >authentication, and we are hoping to make that a standard in > OPenDAP > >clients, which can act like browsers and pop up a > username/password > >dialog window, instead of embedding the user:pass@ in the URL. > > > >>Questions > >>--------------- > >>We have the following questions: > >>1) Where in the netCDF API would the content manager that handles > >>GRD georeferencing information sit? > >>2) How does the I/O SP architecture determine the I/O SP for a given > >>file:// <file://\> style URL? How would it know to handle a > grd:// URL > >>differently? > > > >Very perceptive question; let me start here to explain these 2 > questions:> > >The IOSP architecture is, in fact (RandomAccessFile) file based. > >Since you will be URL based, we have to fit you in at a higher > >level, namely NetcdfDataset.openFile(). If you look there you will > >see that we look for opendap (http: or dods:) and thredds: URLs. > It > >might makes sense to generalize this to allow plugging in external > >handlers for your protocol, similar to how java.net.ContentHandler > >works. Otherwise we might put your code in the core, which is also > a > >possibility. > > > >Anyway, NetcdfDataset.openFile() would detect your URL scheme and > >call NetcdfFile with your IOSP. We will have to add a new > >constructor for that. (You could alternately just subclass > >NetcdfFile, which is what DODSNetcdfFile does). > > > >As for the "content manager that handles GRD georeferencing > >information". It could be a CoordSysBuilder subclass. However, > this > >is actually unnecessary if you use an existing Convention, and we > >would highly recommend using the CF Convention for gridded data. > >Since you are creating the "file", you can add the attributes and > >variables needed by that Convention. This makes your data "CF > >compliant" automatically, which is a real win. > > > >>3) Have we interpreted the slide on page 53 correctly -- is there > a > >>server that can serve out data using the CDM (via the Java netCDF > >>API) as an intermediate step? > > > >yes, the THREDDS Data Server > > > >>4) Does a group structure to represent GRD contents map to an > >>OPeNDAP connection, WCS, or netCDF file or do those types of data > >>representations only have netCDF variables and no groups? > > > >In principle you could use Groups, but they really wont be fully > >supported until we get the netcdf-4 file format finished and > tested. > >I would advise to start with the simpler case of no groups. > > > >>5) Our understanding of the netCDF Java library is that it has, > in > >>particular, the following two entry points: > >> * NetcdfFile : this is the bare netCDF access to files of > various>> types. It doesn't understand anything about > coordinate systems. > >> You can add an I/O service provider to handle your > favorite file > >> format via a class method. The variables it returns are > instances>> of Variable (which of course don't know anything > about coordinate > >> systems). > >> * NetcdfDataset : this is a layer built above the NetcdfFile > layer>> and is the usual interface for applications (e.g., a > WCS). It > >> handles converting various attributes into a coordinate > system. It > >> has a number of methods relating to adding or getting > coordinate>> systems. These methods seem to be applied to the > entire file, > >> rather than to individual variables (or groups). > > > >coordinate systems are really variable-specific. however the > common > >case is that each dataset has a single coordinate system (or a set > >of closely related ones). > > > > > >> CoordinateSystem > >> > >><" target="l">http://www.unidata.ucar.edu/software/netcdf- > java/v2.2.18/javadoc/ucar/nc2/dataset/CoordinateSystem.html>>> > *findCoordinateSystem*>> > >><" target="l">http://www.unidata.ucar.edu/software/netcdf- > java/v2.2.18/javadoc/ucar/nc2/dataset/NetcdfDataset.html#findCoordinateSystem% 28java.lang.String%29>(>> java.lang.String name) // Retrieve the CoordinateSystem > >> with the specified name. > >> java.util.List *getCoordinateAxes* > >> <" target="l">http://www.unidata.ucar.edu/software/netcdf- > java/v2.2.18/javadoc/ucar/nc2/dataset/NetcdfDataset.html#getCoordinateAxes%28%29> () > >> > >> // Get the list of all CoordinateAxis objects used by > >> this dataset. > >> java.util.List * getCoordinateTransforms * > >> <" target="l">http://www.unidata.ucar.edu/software/netcdf- > java/v2.2.18/javadoc/ucar/nc2/dataset/NetcdfDataset.html#getCoordinateTransforms% 28%29> > >> () > >> // Get the list of all CoordinateTransform objects > used > >> by this dataset. > >> boolean * getCoordSysWereAdded * > >> <" target="l">http://www.unidata.ucar.edu/software/netcdf- > java/v2.2.18/javadoc/ucar/nc2/dataset/NetcdfDataset.html#getCoordSysWereAdded%28% 29> > >> () > >> // Has Coordinate System metadata been added. > >>The NetcdfDataset object contains instances of VariableDS. They > are > >>like a wrapper for the Variable objects found in the NetcdfFile > >>object. There is a method to ask a VariableDS for the list of > >>coordinate systems associated with it. > > > >exactly > > > >>If we interpret things correctly , when a NetcdfDataset object is > >>built from a NetcdfFile object, the NetcdfDataset object is > >>responsible for figuring out the coordinate system information > from > >>attributes in the NetcdfFile, and composing a VariableDS from the > >>coordinate system information and each Variable. In theory, by > >>implementing our own CoordSysBuilder class and registering it, we > >>should be able to add coordinate system information to each > >>VariableDS individually. > > > >yes, or as i mentioned use an existing Convention and > CoordSysBuilder.> > > > >>A question then is : do applications like the web coverage server > >>and OPeNDAP server get their coordinate information from > VariableDS > >>objects or from the NetcdfDataset object? > > > > > >OPenDAP is (more or less) at the same level as NetcdfFile, and so > >just faithfully transmits Variables, Attributes, and Dimensions > >across the wire. The coordinate systems then are added by clients > >(like CDM) that understand the convention. We are expecting that > >DAP4, the future opendap protocol, will add Groups. > > > >WCS, OTOH, works at the coordinate system level, and so uses the > >GridDatatype, which is specialized for "coverage" data, and gets > its > >coordinates systems from NetcdfDataset. The clent makes requests > in > >coordinate space, and we know how to translate that into index > >space. Currently we can send back either geoTiff or netcdf/CF > files. > >There are some limittions- the grid spacing must be uniform in WCS > >1.0. We expect to move to WCS 1.1 later this year, which removes > >that limitation. We havent implemented reprojection/resampling, > and > >im not sure that we will. > > > >>If it is from the NetcdfDataset object, then the strategy of > >>grouping all the grids in a database into a single NetcdfDataset, > >>as outline above, won't work, and we'd be obliged to use a > THREDDS > >>server. Is this correct? > > > >It would likely be a mistake to put a lot of disparate data into > the > >same NetcdfDataset. Better to find the right granularity, which is > >typically homogenous data that shares the same discovery > >metadata. So I would not use the Group mechanism to break the > data > >into granules, better to make seperate datasets. Its possible that > >such an idiom will develop with Netcdf-4, but better to get > >something working that stays within existing practice, then decide > >if you want to forge ahead. Let me emphasize that its really > >important to find the right dataset granularity. > > > >This means you want to use THREDDS catalogs to publish the dataset > >URLs and associated metadata, and possibly use TDS to serve your > >data. Once you had an IOSP or equivilent for your data, the main > >work is to develop the catalogs. These can be pretty minimal, but > >automatically populating catalogs with high-quality metadata is a > >huge win in the long run. > > > >I think that would be a powerful value-added product, but of > course > >i dont know what your customers really want. As Ted mentioned, its > a > >good time to help influence TDS strategy, and it appears to me > that > >your small company with extensive scientific experience would be a > >good fit with Unidata. > > > >John > > ********************************************** > Ian Barrodale, Ph.D. > President > Barrodale Computing Services Ltd. > Tel: (250) 472-4372 Fax: (250) 472-4373 > Web: http://www.barrodale.com > Email: ian@xxxxxxxxxxxxx > ********************************************** > Mailing Address: > P.O. Box 3075 STN CSC > Victoria BC Canada V8W 3W2 > > Shipping Address: > Hut R, McKenzie Avenue > University of Victoria > Victoria BC Canada V8W 3W2 > ********************************************** > >
netcdf-java archives: