CrawlableDataset and Catalog Generation

thredds.crawlabledataset.CrawlableDataset

The CrawlableDataset interface was designed to be an abstract representation of a hierarchical collection of datasets. The goal was to allow various data storage systems to be mapped into the CrawlableDataset model and from there used to generate THREDDS catalogs and provide access to datasets so they can be served by the TDS.

public interface CrawlableDataset {
  public String getPath();
  public String getName();

  public boolean exists();

  public CrawlableDataset getParentDataset();

  public boolean isCollection();
  public CrawlableDataset getDescendant( String relativePath);
  public List listDatasets() throws IOException;
  public List listDatasets( CrawlableDatasetFilter filter ) throws IOException;

  public long length();
  public Date lastModified();
}

It is a very simple interface similar to (and based upon) java.io.File. The reference implementation, CrawlableDatasetFile, is built on top of java.io.File. Two other implementations exist:

• the first one implements CrawlableDataset on top of the OPeNDAP server dods_dir responses;
• the second one implements CrawlableDataset on top of the OPeNDAP Server4 BES responses.

• Crawling remote THREDDS catalogs (will be useful when NcML aggregation uses CrawlableDataset for scanning).
• Using a CrawlableDataset implementation to reorganizing an existing CrawlableDataset collection.
• Other ideas? Let us know.
  

thredds.cataloggen.CatalogBuilder

The CatalogBuilder interface provides methods for generating THREDDS catalogs. The main implementation generates catalogs based on the datasetScan configuration

The Goal

The goal for the CrawlableDataset and CatalogBuilder family of interfaces (see diagram below) is to allow users to change how catalogs are built, beyond what is currently available in the datasetScan configuration files.

More Detail

CrawlableDataset and CatalogBuilder Architecture

This document is maintained by Unidata and was last updated July 2007. Send comments to THREDDS support.