[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Catalog generator crawling remote OPenDAP/DODS servers



Hi Valentijn,

Valentijn Venus wrote:

Hi Ethan,

First a fundamental questions related to THREDDS: the idea for the
catalog generator is to crawl whatever source of data you have (local or
remote) and generate a catalog, correct?

That is the idea but unfortunately crawling a local file system is all that is currently implemented.

Now I'm looking for some
directions on how to explore all sub-directories on a remote DODS
server. The following servers I have in mind:

MODIS AQUA/TERRA:
http://g0dup05u.ecs.nasa.gov/OPENDAP_DP/long_term/MOAA/MYD04_L2.004/2002
.07.16/
TOMS Ozone:
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/T
OMS/EP/


Recently they moved these servers, and since then only the DODS Dataset
Access Form for TOMS seems to work. When accessing one of these HDF
files (i.e.
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/T
OMS/EP/TOMS-EP_L3-TOMSEPL3_1996m0725_v8.HDF.html) and once the form is
setup, I can see the resulting "get"/"post" parameters trailing the "?"
in a DODS url (...?Ozone[0:1:179][0:1:287]) How does this work once the
catalog is generated, and a client (IDV, Matlab, etc.) accesses it in
real-time? I guess THREDDS supports geographic/paramter subsetting in a
similair manner, but how does it work technically?


Generally, the THREDDS catalog provides the base DODS URL and the client needs to understand the OPeNDAP/DODS protocol (the ".dds", ".das", ".dods" extensions and all the stuff after the "?"). THREDDS allows for metadata to indicate the geographic coverage of a dataset but it doesn't support subsetting a particular dataset. The actual subsetting all dependent on the protocol being used to access the dataset, in this case OPeNDAP.


Back to the catalog generation. At one point I did have OPeNDAP server crawling capabilities in the catalog generator but no one was using it and I don't think it is still working after a few major modifications. Are you using the stand-alone CatalogGen application, the catalog generation servlet interface, or the newer TDS configuration which also generates catalogs? The TDS config is a completely different framework than that used by the app and servlet. The code (non-TDS framework) for crawling OPeNDAP servers is still around if you want to look into getting it running again. However, I'm planning on moving the CatGen app and servlet over to the new TDS framework at some point but that may be awhile.


Hope this helps.

Ethan

Cheers, Valentijn



-- Ethan R. Davis Telephone: (303) 497-8155 Software Engineer Fax: (303) 497-8690 UCAR Unidata Program Center E-mail: address@hidden P.O. Box 3000 Boulder, CO 80307-3000 http://www.unidata.ucar.edu/ ---------------------------------------------------------------------------



NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.