TDS Tutorial: TDS Configuration

Example Catalog

  <?xml version='1.0' encoding='UTF-8'?>
  <catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
           xmlns:xlink="http://www.w3.org/1999/xlink"
           version="1.0.2">
1)    <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
      <dataset name="TDS Tutorial: example 1">

3)        <dataset name="TDS Tutorial: example data 1" urlPath="test/example1.nc" >
2)            <serviceName>odap</serviceName>
          </dataset>
          <dataset name="TDS Tutorial: example data 2" urlPath="test/example2.nc" >
              <serviceName>odap</serviceName>
          </dataset>
          <dataset name="TDS Tutorial: example data 3" urlPath="test/example3.nc" >
              <serviceName>odap</serviceName>
          </dataset>

          <catalogRef xlink:title="My Other Catalog"
                      xlink:href="myOtherCatalog.xml" />
          <catalogRef xlink:title="Far Away Univ catalog"
                      xlink:href="http://www.farAwayU.edu/thredds/catalog.xml" />
      </dataset>
  </catalog>

Notes:

• The service element defines an OPeNDAP service and has the name "odap".
• The first dataset is a container dataset.
• Each child dataset has an access method due to its urlPath attribute and child serviceName element.
• The catalogRef elements link to a relative catalog and a remote catalog.

Constructing an access URL

1. Find the service element referenced by the dataset:

   <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />

2. Get the base URL of the service element:

   /thredds/dodsC/

3. If it is a relative URL, resolve against the catalog's URL:

   http://my.server/thredds/catalog.xml      // Catalog URL

   http://my.server/thredds/dodsC/           // Service base URL

4. Find the urlPath attribute for the accessible dataset:

   urlPath="test/example3.nc"

5. Append the value of the urlPath attribute to the base service URL:

   http://my.server/thredds/dodsC/test/example3.nc

More information is available on constructing access URLs.

Example Catalog - shared service name

  <?xml version='1.0' encoding='UTF-8'?>
  <catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
           xmlns:xlink="http://www.w3.org/1999/xlink"
           version="1.0.2">
      <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
      <dataset name="TDS Tutorial: example 2">
          <metadata inherited="true">
2)             <serviceName>odap</serviceName>
          </metadata>

3)        <dataset name="TDS Tutorial: example data 1" urlPath="test/example1.nc" />
3)        <dataset name="TDS Tutorial: example data 2" urlPath="test/example2.nc" />
3)        <dataset name="TDS Tutorial: example data 3" urlPath="test/example3.nc" />

          <catalogRef xlink:title="My Other Catalog"
                      xlink:href="myOtherCatalog.xml" />
          <catalogRef xlink:title="Far Away Univ catalog"
                      xlink:href="http://www.farAwayU.edu/thredds/catalog.xml" />
      </dataset>
  </catalog>

Notes:

• The serviceName element is inherited by each child dataset from the containing dataset.

TDS configuration catalogs are THREDDS catalogs with a few extensions. They contain information detailing the datasets the TDS will serve and what services will be available for each dataset:

• The datasetRoot and datasetScan elements are extensions that:
  • provide mappings between incoming URL requests and directories on disk; and
  • are used in the detailing of the datasets the TDS will serve.
• Available services are indicated in the normal THREDDS catalog manner with service name references.

The TDS configuration catalogs represent the top-level catalogs served by the TDS:

• The configuration information is only needed by the server.
• The client view of the catalogs does not contain any configuration information.

datasetRoot Element

Each datasetRoot element defines a single mapping between a URL base path and a directory. The URL base path so defined can then be used in accessible datasets for files under the mapped directory. For instance, if you have a directory /my/data/testdata that contains:

afile.nc
testData.nc
junk.nc
grib/
  data1.grib2
  data2.grib2

You can serve the "testData.nc" and "grib1/data1.grib2" files with the following:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

1)    <datasetRoot path="my/test" location="/my/data/testdata/" />

2)    <dataset name="A Test Dataset" ID="testDataset" urlPath="my/test/testData.nc" >
        <serviceName>odap</serviceName>
      </dataset>
3)    <dataset name="A Test Dataset 2" ID="testDataset2" urlPath="my/test/grib/data1.grib2" >
        <serviceName>odap</serviceName>
      </dataset>
...

The datasetRoot element above (1) maps the "my/test" path to the "/my/data/testdata/" directory. The URLs to access the datasets (2 & 3) are

http://my.server/thredds/dodsC/my/test/testData.nc 
http://my.server/thredds/dodsC/my/test/grib/data1.grib2

When the server receives a request for one of the above URLs, it uses the URL path to look for a matching dataset root. In this case it finds the mapping provided by the datasetRoot element above and looks in the /my/data/testdata directory for the file.

The client catalog that results from this catalog is the same as the above without the datasetRoot element.

datasetScan Element

Each datasetScan element also defines a single mapping between a URL base path and a directory. Unlike the  datasetRoot element which works with dataset elements to define the datasets served, the datasetScan element will automatically serve some or all of the datasets found in the mapped directory. So, all the files in the above listing could be served with the following:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

2)    <datasetScan name="Test all files in a directory" ID="testDatasetScan" 
                   path="my/test/all" location="/my/data/testdata">
          <metadata inherited="true">
              <serviceName>odap</serviceName>
          </metadata>
      </datasetScan>
...

In the client view of a configuration catalog, datasetScan elements are converted to catalogRef elements. So, the resulting client view of this catalog looks like:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

2)    <catalogRef xlink:title="Test all files in a directory" ID="testDatasetScan" 
                  xlink:href="/thredds/catalog/my/test/all/catalog.xml" name="" />
...

The generation of the catalog referenced by the catalogRef element is deferred until a request is made for that catalog. When the catalog is requested the location directory is scanned, directories are represented as catalogRef elements and files are represented as dataset elements. The scanning of each subdirectory is deferred until a request is made for the corresponding catalog. The catalog referenced above would look like:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

      <dataset name="Test all files in a directory" ID="testDatasetScan" >
          <metadata inherited="true">
              <serviceName>odap</serviceName>
          </metadata>
          <dataset name="afile.nc" ID="testDatasetScan/afile.nc" urlPath="my/test/all/afile.nc">
          <dataset name="testData.nc" ID="testDatasetScan/testData.nc" urlPath="my/test/all/testData.nc">
          <dataset name="junk.nc" ID="testDatasetScan/junk.nc" urlPath="my/test/all/junk.nc">

          <catalogRef xlink:title="grib" ID="testDatasetScan/grib" name=""
                      xlink:href="/thredds/catalog/my/test/all/grib/catalog.xml" />
      </dataset>
...

Note: The datasetScan element provides for limiting the datasets that are included in the scan, changing the names of datasets, sorting datasets, etc. We will go into more detail on this later.

TDS Configuration Directory

All TDS configuration information is stored in the TDS configuration directory. Its default location is ${TOMCAT_HOME}/content/thredds and it is created and populated with default files the first time the TDS is deployed. It is persistent even when a TDS installation is upgraded or re-deployed. All your configuration, modifications, and additions should be made in this directory. Do not place files containing passwords or anything else with security issues in this directory. Typically you will only be adding and modifying catalogs and configuration files.

For now, we will focus on the following subset of the content directory:

• ${TOMCAT_HOME}/content/thredds/
  • catalog.xml - the main TDS configuration catalog (root catalog for TDS configuration)
  • enhancedCatalog.xml - an example catalog [Note: It is referenced from catalog.xml.]
  • threddsConfig.xml - configuration file for allowing non-default services, configuring caching, etc. (More on this in the TDS Reference Docs.)
  • logs/
    • catalogInit.log - log file for messages generated while reading TDS configuration catalogs during TDS initialization and reinitialization.
    • threddsServlet.log - TDS error log file.

TDS Root Catalog

The main TDS configuration catalog is at ${TOMCAT_HOME}/content/thredds/catalog.xml. We ship a simple test catalog:

  <?xml version="1.0" encoding="UTF-8"?>
  <catalog name="THREDDS Server Default Catalog : You must change this to fit your server!"
           xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
           xmlns:xlink="http://www.w3.org/1999/xlink" >
      <service name="thisDODS" serviceType="OpenDAP" base="/thredds/dodsC/" />

1)    <datasetRoot path="test" location="content/testdata/" />

      <dataset name="Test Single Dataset" ID="testDataset"
               serviceName="thisDODS" urlPath="test/testData.nc" />

      <datasetScan name="Test all files in a directory" ID="testDatasetScan"
1)             path="testAll" location="content/testdata">
          <metadata inherited="true">
              <serviceName>thisDODS</serviceName>
          </metadata>

          <filter>
2)           <include wildcard="*.nc" />
          </filter>

      </datasetScan>
    
3)    <catalogRef xlink:title="Test Enhanced Catalog"
                  xlink:href="enhancedCatalog.xml" name ="" />
    </catalog>

When the TDS starts, this main configuration catalog is read as are all catalogs in the catalog tree defined by catalogRef elements. The TDS gathers configuration information on what datasets it should serve from the datasetRoot and datasetScan elements.