[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: CrawlableDatasetDods II



Hi Bas,

Sorry again for the delay. I've been out of the office a fair amount this summer.

Bas Retsios wrote:

So far so good, I got something working. It was a matter of getting the catalog.xml file right. At this moment, I get the job done, but I would like you to check if this is the optimal way.

Glad to hear things are working.


The java code of CrawlableDatasetDods did not change up to now (it is the same as the one I sent you on June 23rd).

What I changed was the datasetScan and the service sections of my catalog.xml file. This is what I needed in order to access DODS server http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/EP/

catalog.xml:
=============================
...
<service name="remoteopendap" serviceType="OpenDAP" base="http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/"/>
...
...
<datasetScan name="DodsTest" path="EP" location="http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/EP"; ID="nasaTest" addDatasetSize="true" addLatest="true"> <crawlableDatasetImpl className="thredds.crawlabledataset.CrawlableDatasetDods" />
   <metadata inherited="true">
     <serviceName>remoteopendap</serviceName>
   </metadata>
 </datasetScan>
...
=============================

Some comments:
1. I see almost the same URL twice in this catalog.xml file: once at <service> and once at <datasetScan>
2. base + path = location, otherwise it does not work

I was wondering if it is possible to avoid having 2 times the same URL. Unfortunately something is checking if "base" is a parent of "location".

I don't believe this is possible at the moment. There are a number of checks to insure consistency between various locations but the "base" and "location" aren't being directly compared. In the file based version, the "base" and "location" are very different. The issue is that you are dealing with datasets that you aren't serving with the TDS so the "base" and the "location" have to overlap to a large degree.

One of the reasons for all these issues is that our main design goal was to allow the server to access local and remote datasets so that the server could then catalog the data services it was providing for those datasets. Your case is cataloging the data services provided by a remote dataset. I certainly was planning on this working and thankfully it seems to be working for you. However, it wasn't really one of our use cases so we didn't test it thoroughly or get to think through all the little design issue that use case would have brought up. I'll have to think about whether there are some nice ways to make your use case a little more elegant.

Furthermore, "path" is the same for my local thredds server and the remote server. In my case, path="EP", thus an example of valid local and remote URLs is: http://thredds.itc.nl:8080/thredds/catalog/EP/catalog.html?dataset=nasaTest/TOMS-EP_L3-TOMSEPL3_1996m0802_v8.HDF http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/EP/TOMS-EP_L3-TOMSEPL3_1996m0802_v8.HDF.html My problem with this is that the string "EP" originates from the remote URL. I was forced to put it as "path" into catalog.xml. As a result, "EP" became part of my local server URLs, while I liked to call it "testDods".

I think you can change the "EP" in your URLs to "testDods" by adding "EP/" to "base" and changing "path" to "testDods". One issue here would be that, if you currently have multiple datasetsScans referencing the same service, this will require multiple service elements each with a slightly different "base".


Well, what I have mentioned up to now does work, but I need to edit my catalog.xml file and restart the server everytime I need to include a new DODS server.

If you have remote configuration setup on your server (see http://motherlode.ucar.edu:8080/thredds/docs/RemoteManagement.html), you can reload the catalog.xml file without restarting Tomcat or the TDS. You just go to the debug page on your server (http://hostname:port/thredds/debug) and select the "Reinitialize" link.

Ideally I would like to have a location on my server that I can use to convert any DODS url to a catalog. A location could be e.g.:
http://thredds.itc.nl:8080/thredds/catalog/dodsToCatalog/
Then after /dodsToCatalog/ I could append any remote URL, for which my server can make a catalog. Is this possible?

That's a great idea but I'm afraid we don't have anything like that at the moment.

Hope this helps. Let me know if you have any other issues or thoughts on how this all works.

Thanks,

Ethan


Cheers,

Bas.


--
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------