[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #QIL-297611]: aggregation of remote datasets CrawlableDatasetDods



Hi Tyn,
 
Valentijn Venus wrote:
> 
> Does this mean that the CrawlableDatasetDods is on your list of TO-DO's?
> Will it allow for remote OPeNDAP servers (without THREDDS) to be
> crawled? And later on ncml-aggregated/enhanced?

I'm not sure I understand what you mean by "without THREDDS".

The following has been supported for some time (since Bas wrote 
CrawlableDatasetDods) and still is supported:

1) The TDS will crawl OPeNDAP servers and generate catalogs using 
CrawlableDatasetDods.

2) CrawlableDatasetDods can be used programatically to crawl OPeNDAP servers.

The following will be available once it is fully implemented (on our ToDo list 
but it hasn't been at the top):

1) Crawling OPeNDAP servers to find datasets from which to build an aggregation.
 
> > thredds.crawlabledataset classes are distributed in the netcdf-4.0.jar
> file in the WEB-INF/lib directory
> 
> In which jar, there are so many there (WEB-INF/lib directory) but non
> carries the name thredds.crawlabledatase or any deriviative ?

In the netcdf-4.0.jar file.
 
> There is a conference in October this year where I would like to present
> some of the tds stuff we have been trying over the last couple of years.
> This feature is certainly something I'd like to include in that
> paper/presentation. I recall vivedly Bas (the programmer who worked with
> yo uat the time) spending more hours to implement this along the new TDS
> framework as-per your request. It would nice to see it working in the
> final release of TDS 4.0.
> 
> Any ideas how to get this going, perhaps Bas can help again?

Again, I'm not sure I'm understanding. Is something that was working no longer 
working?

What feature is it that you want to get going?

Sorry if I'm being a bit dense.

Ethan

> tyn
 
> -----Original Message-----
> From: Unidata THREDDS Support [mailto:address@hidden
> Sent: Monday, May 11, 2009 11:47 PM
> To: Valentijn Venus
> Subject: [THREDDS #QIL-297611]: aggregation of remote datasets
> CrawlableDatasetDods
> 
> Hi Tyn,
> 
> Valentijn Venus wrote:
> > Any news on the crawlable dataset side and adding aggregation support
> > for it?
> 
> Aggregation now uses CrawlableDataset under the hood. However, its
> default implementation is CrawlableDatasetFile. There are some hooks in
> the code (ucar.nc2.ncml.Aggregation and .DatasetScanner) to allow for
> other CrawlableDataset implementations but as yet the link between the
> reading of the NcML and those hooks doesn't exist.
> 
> > I just downloaded TDS vesion 4 but cannot find the
> > CrawlableDatasetDods class we contributed. Is this class no longer
> > supported as I can see the package classes/thredds/crawlabledataset is
> removed all together?
> 
> All the thredds.crawlabledataset classes are distributed in the
> netcdf-4.0.jar file in the WEB-INF/lib directory of the thredds.war
> file.
> 
> Ethan
> 
> > files under thredds/crawlabledataset:
> >
> > CrawlableDatset.java
> > CrawlableDatasetAlias.java
> > CrawlableDatsetDods.java    <-- we're looking for this
> > CrawlableDatsetFactory.java
> > CrawlableDatsetFile.java
> > CrawlableDatsetFilter.java
> > CrawlableDatsetLabeler.java
> > CrawlableDatsetSorter.java
> > MultiLabeler.java
> > RegExpAndReplaceOnNameLabeler.java
> > RegExpAndReplaceOnPathLabeler.java
> >
> > We hope it will be back in the final release?
> >
> > Cheers, Tyn


Ticket Details
===================
Ticket ID: QIL-297611
Department: Support THREDDS
Priority: Normal
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.