[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Catalog generator crawling remote OPenDAP/DODS servers

Hi Valentijn,

Actually, the CatalogGen servlet is included as part of the TDS. If you are writing configuration files that include <datasetScan> elements, you are using the newer TDS configuration. The URLs for this stuff are of this form: "http://<server>:<port>/thredds/...". If you are writing configuration files that include <catalogGenConfig> elements, you are using the old catGen framework. The CatalogGen servlet is at "http://<server>:<port>/thredds/cataloggen/...".

The 3.8 code (and now the 3.10 code) are available at


The code for both the old and new style catalog generation are included. The old implementation for crawling OPeNDAP/DODS servers is in src/thredds/cataloggen/config/DodsDirDatasetSource.java. In the new framework, an implementation of src/thredds/crawlabledataset/CrawlableDataset.java would have to be written to handle crawling an OPeNDAP/DODS server. The old stuff might be easy to get up and running quickly but since the plan is to move everything to the CrawlableDataset framework, I would encourage you to go that route.

Let me know what you decide.


Valentijn Venus wrote:
Thanks for you response. I guess with thredds.war 3.8 i'm using the newer TDS 
configuration which also generates catalogs?

Can you send me a pointer to that old code please? Thanks again...


-----Original Message-----
From: Ethan Davis [mailto:address@hidden
Sent: Mon 6/5/2006 19:58
To: Valentijn Venus
Cc: address@hidden
Subject: Re: Catalog generator crawling remote OPenDAP/DODS servers
Hi Valentijn,

Valentijn Venus wrote:

Hi Ethan,

First a fundamental questions related to THREDDS: the idea for the
catalog generator is to crawl whatever source of data you have (local or
remote) and generate a catalog, correct?

That is the idea but unfortunately crawling a local file system is all that is currently implemented.

Now I'm looking for some
directions on how to explore all sub-directories on a remote DODS
server. The following servers I have in mind:

TOMS Ozone:

Recently they moved these servers, and since then only the DODS Dataset
Access Form for TOMS seems to work. When accessing one of these HDF
files (i.e.
OMS/EP/TOMS-EP_L3-TOMSEPL3_1996m0725_v8.HDF.html) and once the form is
setup, I can see the resulting "get"/"post" parameters trailing the "?"
in a DODS url (...?Ozone[0:1:179][0:1:287]) How does this work once the
catalog is generated, and a client (IDV, Matlab, etc.) accesses it in
real-time? I guess THREDDS supports geographic/paramter subsetting in a
similair manner, but how does it work technically?

Generally, the THREDDS catalog provides the base DODS URL and the client needs to understand the OPeNDAP/DODS protocol (the ".dds", ".das", ".dods" extensions and all the stuff after the "?"). THREDDS allows for metadata to indicate the geographic coverage of a dataset but it doesn't support subsetting a particular dataset. The actual subsetting all dependent on the protocol being used to access the dataset, in this case OPeNDAP.

Back to the catalog generation. At one point I did have OPeNDAP server crawling capabilities in the catalog generator but no one was using it and I don't think it is still working after a few major modifications. Are you using the stand-alone CatalogGen application, the catalog generation servlet interface, or the newer TDS configuration which also generates catalogs? The TDS config is a completely different framework than that used by the app and servlet. The code (non-TDS framework) for crawling OPeNDAP servers is still around if you want to look into getting it running again. However, I'm planning on moving the CatGen app and servlet over to the new TDS framework at some point but that may be awhile.

Hope this helps.


Cheers, Valentijn

-- Ethan R. Davis Telephone: (303) 497-8155 Software Engineer Fax: (303) 497-8690 UCAR Unidata Program Center E-mail: address@hidden P.O. Box 3000 Boulder, CO 80307-3000 http://www.unidata.ucar.edu/ ---------------------------------------------------------------------------

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.