THREDDS Configuration Catalogs

TDS Configuration Directory Layout

Location of the TDS Configuration Directory

If desired, the location can be set to a directory outside of the Tomcat install directory. See the TDS Configuration Directory section of the TDS Reference for more information.

All TDS configuration information is stored in the TDS configuration directory. Its default location is ${tomcat_home}/content/thredds and it is created and populated with default files the first time the TDS is deployed. It is persistent even when a TDS installation is upgraded or re-deployed. All your configuration, modifications, and additions should be made in this directory. Do not place files containing passwords or anything else with security issues in this directory. Typically you will only be adding and modifying catalogs and configuration files.

For now, we will focus on the following subset of the content directory:

  • ${tomcat_home}/content/thredds/
    • catalog.xml - the main TDS configuration catalog (root catalog for TDS configuration)
    • enhancedCatalog.xml - an example catalog [Note: It is referenced from catalog.xml.]
    • threddsConfig.xml - configuration file for allowing non-default services, configuring caching, etc (more details available here).
    • logs/
      • catalogInit.log - log file for messages generated while reading TDS configuration catalogs during TDS initialization and reinitialization.
      • threddsServlet.log - log messages about individual TDS requests, including any error messages. Useful for debugging problems.
    • cache/ - various cache directories
      • agg/
      • cdm/
      • collection/
      • ehcache/
      • ncss/
      • wcs/

TDS Root Catalog

The main TDS configuration catalog is at ${tomcat_home}/content/thredds/catalog.xml. We ship a simple test catalog:

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="THREDDS Server Default Catalog : You must change this to fit your server!"
        xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
        xmlns:xlink="http://www.w3.org/1999/xlink">

  <service name="all" base="" serviceType="compound">
    <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
    <!--service name="wms" serviceType="WMS" base="/thredds/wms/" /-->
    <!--service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" /-->
  </service>

  <datasetRoot path="test" location="content/testdata/" />

  <dataset name="Test Single Dataset" ID="testDataset" serviceName="odap"
           urlPath="test/testData.nc" dataType="Grid"/>

  <dataset name="Test Single Dataset 2" ID="testDataset2" serviceName="odap" urlPath="test/testData2.grib2" dataType="Grid"/>

  <datasetScan name="Test all files in a directory" ID="testDatasetScan" path="testAll" location="content/testdata">

    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>Grid</dataType>
    </metadata>

    <filter>
      <include wildcard="*eta_211.nc"/>
    </filter>

  </datasetScan>

  <catalogRef xlink:title="Test Enhanced Catalog" xlink:href="enhancedCatalog.xml" name=""/>

</catalog>

When the TDS starts, this main configuration catalog is read as are all catalogs in the catalog tree defined by catalogRef elements. The TDS gathers configuration information on what datasets it should serve from the datasetRoot and datasetScan elements.

TDS Configuration Catalogs

TDS configuration catalogs are like THREDDS Client catalogs with a few extensions. They contain information detailing the datasets the TDS will serve and what services will be available for each dataset:

  • The datasetRoot and datasetScan elements are extensions that:
    • provide mappings between incoming URL requests and directories on disk; and
    • are used in the detailing of the datasets the TDS will serve.
  • Available services are indicated in the normal THREDDS catalog manner with service name references.

The TDS configuration catalogs represent the top-level catalogs served by the TDS:

  • The configuration information is only needed by the server.
  • The client view of the catalogs does not contain any configuration information.

datasetRoot Element

Each datasetRoot element defines a single mapping between a URL base path and a directory. The URL base path so defined can then be used in accessible datasets for files under the mapped directory. For instance, if you have a directory /my/data/testdata that contains:

afile.nc
testData.nc
junk.nc
grib/
  data1.grib2
  data2.grib2

You can serve the "testData.nc" and "grib1/data1.grib2" files with the following:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

1)    <datasetRoot path="my/test" location="/my/data/testdata/" />

2)    <dataset name="A Test Dataset" ID="testDataset" urlPath="my/test/testData.nc" >
        <serviceName>odap</serviceName>
      </dataset>
3)    <dataset name="A Test Dataset 2" ID="testDataset2" urlPath="my/test/grib/data1.grib2" >
        <serviceName>odap</serviceName>
      </dataset>
...

The datasetRoot element above (1) maps the "my/test" path to the "/my/data/testdata/" directory. The URLs to access the datasets (2 & 3) are

http://hostname:port/thredds/dodsC/my/test/testData.nc
http://hostname:port/thredds/dodsC/my/test/grib/data1.grib2

When the server receives a request for one of the above URLs, it uses the URL path to look for a matching dataset root. In this case it finds the mapping provided by the datasetRoot element above and looks in the /my/data/testdata directory for the file.

The client catalog that results from this catalog is the same as the above without the datasetRoot element.

datasetScan Element

Each datasetScan element also defines a single mapping between a URL base path and a directory. Unlike the datasetRoot element which works with dataset elements to define the datasets served, the datasetScan element will automatically serve some or all of the datasets found in the mapped directory. So, all the files in the above listing could be served with the following:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

      <datasetScan name="Test all files in a directory" ID="testDatasetScan"
                   path="my/test/all" location="/my/data/testdata">
        <metadata inherited="true">
          <serviceName>odap</serviceName>
        </metadata>
      </datasetScan>
...

In the client view of a configuration catalog, datasetScan elements are converted to catalogRef elements. So, the resulting client view of this catalog looks like:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

      <catalogRef xlink:title="Test all files in a directory" ID="testDatasetScan"
                  xlink:href="/thredds/catalog/my/test/all/catalog.xml" name="" />
...

The generation of the catalog referenced by the catalogRef element is deferred until a request is made for that catalog. When the catalog is requested the location directory is scanned, directories are represented as catalogRef elements and files are represented as dataset elements. The scanning of each subdirectory is deferred until a request is made for the corresponding catalog. The catalog referenced above would look like:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

      <dataset name="Test all files in a directory" ID="testDatasetScan" >
        <metadata inherited="true">
          <serviceName>odap</serviceName>
        </metadata>
        <dataset name="afile.nc" ID="testDatasetScan/afile.nc" urlPath="my/test/all/afile.nc">
        <dataset name="testData.nc" ID="testDatasetScan/testData.nc" 
                 urlPath="my/test/all/testData.nc">
        <dataset name="junk.nc" ID="testDatasetScan/junk.nc" urlPath="my/test/all/junk.nc">

        <catalogRef xlink:title="grib" ID="testDatasetScan/grib" name=""
                      xlink:href="/thredds/catalog/my/test/all/grib/catalog.xml" />
      </dataset>
...

Note: The datasetScan element provides ways for limiting the datasets that are included in the scan, changing the names of datasets, sorting datasets, etc. We will go into more detail on this later.

Exercise: Add NCEP NAM model data

Modify the main TDS configuration catalog to include some model data.

  1. Take a look at the data (note: the filenames may be different):
    $ ls /machine/tds/data/ldm
    fsl  madis  nam_12km  suomi
    $ ls /machine/tds/data/ldm/nam_12km
    201307211800.grib2  201307220600.grib2  201307221800.grib2  201307230600.grib2
    201307220000.grib2  201307221200.grib2  201307230000.grib2  201307231200.grib2
    
  2. Edit the main TDS configuration catalog:
    $ cd ${tomcat_home}/content/thredds
    $ vi catalog.xml         // Use the editor of your choice
    
  3. And add a datasetScan element for the NAM data:
    <datasetScan name="NCEP NAM 12km" ID="NAM_12km"
                 path="nam_12km" location="/machine/tds/data/ldm/nam_12km">
      <metadata inherited="true">
        <serviceName>odap</serviceName>
      </metadata>
    </datasetScan>
    
  4. Restart Tomcat so the TDS is reinitialized:
    $ cd ${tomcat_home}/bin
    $ ./shutdown.sh
    $ ./startup.sh
    
  5. Test that the new datasetScan is working:
    1. Bring the catalog up in a browser: http://localhost:8080/thredds/catalog.html
    2. Click down to one of the NAM dataset pages.
    3. Select the OPeNDAP link

Not working?

  1. Take a look at the catalogInit.log:
    $ cd ${tomcat_home}/content/thredds/logs
    $ more catalogInit.log
    
  2. Take a look at the threddsServlet.log:
    $ cd ${tomcat_home}/content/thredds/logs
    $ more threddsServlet.log