Basic TDS Configuration

THREDDS Catalog Review

THREDDS catalogs collect, organize, and describe accessible datasets. They provide a hierarchical structure for organizing the datasets as well as an access method (URL) and a human understandable name for each dataset. Further descriptive information about each dataset can also be added.

Example Catalog

  <?xml version='1.0' encoding='UTF-8'?>
  <catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
           xmlns:xlink="http://www.w3.org/1999/xlink"
           version="1.0.2">
1)    <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
2)      <dataset name="TDS Tutorial: example 1">

3)        <dataset name="TDS Tutorial: example data 1" urlPath="test/example1.nc" >
            <serviceName>odap</serviceName>
          </dataset>
          <dataset name="TDS Tutorial: example data 2" urlPath="test/example2.nc" >
              <serviceName>odap</serviceName>
          </dataset>
          <dataset name="TDS Tutorial: example data 3" urlPath="test/example3.nc" >
              <serviceName>odap</serviceName>
          </dataset>

4)        <catalogRef xlink:title="My Other Catalog"
                      xlink:href="myOtherCatalog.xml" />
          <catalogRef xlink:title="Far Away Univ catalog"
                      xlink:href="http://www.farAwayU.edu/thredds/catalog.xml" />
      </dataset>
  </catalog>

Notes

1) The service element defines an OPeNDAP service and has the name "odap".

2) The first dataset is a container dataset.

3) Each child dataset has an access method due to its urlPath attribute and child serviceName element.

4) The catalogRef elements link to a relative catalog and a remote catalog.

Constructing an access URL

Find the service element referenced by the dataset:

<service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />

Get the base URL of the service element:
```
/thredds/dodsC/
```

If it is a relative URL, use the catalog's URL:

http://hostname:port/thredds/catalog.xml      // Catalog URL

to resolve the base URL:

http://hostname:port/thredds/dodsC/           // Service base URL

Find the urlPath attribute for the accessible dataset:

<dataset name="TDS Tutorial: example data 3" urlPath="test/example3.nc" >

Append the value of the urlPath attribute to the base service URL:

http://hostname:port/thredds/dodsC/test/example3.nc
|                   |             |                |
|<---  server   --->|<----------->|<-------------->|
                            |             |
          service basePath -|             |- dataset urlPath

More information is available on constructing access URLs.

Example Catalog - shared service name

  <?xml version='1.0' encoding='UTF-8'?>
  <catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
           xmlns:xlink="http://www.w3.org/1999/xlink"
           version="1.0.2">
      <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
      <dataset name="TDS Tutorial: example 2">
          <metadata inherited="true">
1)             <serviceName>odap</serviceName>
          </metadata>

2)        <dataset name="TDS Tutorial: example data 1" urlPath="test/example1.nc" />
2)        <dataset name="TDS Tutorial: example data 2" urlPath="test/example2.nc" />
2)        <dataset name="TDS Tutorial: example data 3" urlPath="test/example3.nc" />

          <catalogRef xlink:title="My Other Catalog"
                      xlink:href="myOtherCatalog.xml" />
          <catalogRef xlink:title="Far Away Univ catalog"
                      xlink:href="http://www.farAwayU.edu/thredds/catalog.xml" />
      </dataset>
  </catalog>

Notes

1) The serviceName element is inherited by each child dataset from the containing dataset.

2) These three child datasets all inherit the odap serviceName element.

TDS Configuration Catalogs

TDS configuration catalogs are THREDDS catalogs with a few extensions. They contain information detailing the datasets the TDS will serve and what services will be available for each dataset:

The datasetRoot and datasetScan elements are extensions that:
- provide mappings between incoming URL requests and directories on disk; and
- are used in the detailing of the datasets the TDS will serve.
Available services are indicated in the normal THREDDS catalog manner with service name references.

The TDS configuration catalogs represent the top-level catalogs served by the TDS:

The configuration information is only needed by the server.
The client view of the catalogs does not contain any configuration information.

`datasetRoot` Element

Each datasetRoot element defines a single mapping between a URL base path and a directory. The URL base path so defined can then be used in accessible datasets for files under the mapped directory. For instance, if you have a directory /my/data/testdata that contains:

afile.nc
testData.nc
junk.nc
grib/
  data1.grib2
  data2.grib2

You can serve the "testData.nc" and "grib1/data1.grib2" files with the following:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

1)    <datasetRoot path="my/test" location="/my/data/testdata/" />

2)    <dataset name="A Test Dataset" ID="testDataset" urlPath="my/test/testData.nc" >
        <serviceName>odap</serviceName>
      </dataset>
3)    <dataset name="A Test Dataset 2" ID="testDataset2" urlPath="my/test/grib/data1.grib2" >
        <serviceName>odap</serviceName>
      </dataset>
...

The datasetRoot element above (1) maps the "my/test" path to the "/my/data/testdata/" directory. The URLs to access the datasets (2 & 3) are

http://hostname:port/thredds/dodsC/my/test/testData.nc
http://hostname:port/thredds/dodsC/my/test/grib/data1.grib2

When the server receives a request for one of the above URLs, it uses the URL path to look for a matching dataset root. In this case it finds the mapping provided by the datasetRoot element above and looks in the /my/data/testdata directory for the file.

The client catalog that results from this catalog is the same as the above without the datasetRoot element.

`datasetScan` Element

Each datasetScan element also defines a single mapping between a URL base path and a directory. Unlike the datasetRoot element which works with dataset elements to define the datasets served, the datasetScan element will automatically serve some or all of the datasets found in the mapped directory. So, all the files in the above listing could be served with the following:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

      <datasetScan name="Test all files in a directory" ID="testDatasetScan"
                   path="my/test/all" location="/my/data/testdata">
          <metadata inherited="true">
              <serviceName>odap</serviceName>
          </metadata>
      </datasetScan>
...

In the client view of a configuration catalog, datasetScan elements are converted to catalogRef elements. So, the resulting client view of this catalog looks like:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

      <catalogRef xlink:title="Test all files in a directory" ID="testDatasetScan"
                  xlink:href="/thredds/catalog/my/test/all/catalog.xml" name="" />
...

The generation of the catalog referenced by the catalogRef element is deferred until a request is made for that catalog. When the catalog is requested the location directory is scanned, directories are represented as catalogRef elements and files are represented as dataset elements. The scanning of each subdirectory is deferred until a request is made for the corresponding catalog. The catalog referenced above would look like:

...
      <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />

      <dataset name="Test all files in a directory" ID="testDatasetScan" >
          <metadata inherited="true">
              <serviceName>odap</serviceName>
          </metadata>
          <dataset name="afile.nc" ID="testDatasetScan/afile.nc" urlPath="my/test/all/afile.nc">
          <dataset name="testData.nc" ID="testDatasetScan/testData.nc" 
                   urlPath="my/test/all/testData.nc">
          <dataset name="junk.nc" ID="testDatasetScan/junk.nc" urlPath="my/test/all/junk.nc">

          <catalogRef xlink:title="grib" ID="testDatasetScan/grib" name=""
                      xlink:href="/thredds/catalog/my/test/all/grib/catalog.xml" />
      </dataset>
...

Note: The datasetScan element provides ways for limiting the datasets that are included in the scan, changing the names of datasets, sorting datasets, etc. We will go into more detail on this later.

TDS Configuration Directory

Location of the TDS Configuration Directory

If desired, the location can be set to a directory outside of the Tomcat install directory. See the TDS Configuration Directory section of the TDS Reference for more information.

All TDS configuration information is stored in the TDS configuration directory. Its default location is ${tomcat_home}/content/thredds and it is created and populated with default files the first time the TDS is deployed. It is persistent even when a TDS installation is upgraded or re-deployed. All your configuration, modifications, and additions should be made in this directory. Do not place files containing passwords or anything else with security issues in this directory. Typically you will only be adding and modifying catalogs and configuration files.

For now, we will focus on the following subset of the content directory:

${tomcat_home}/content/thredds/
- catalog.xml - the main TDS configuration catalog (root catalog for TDS configuration)
- enhancedCatalog.xml - an example catalog [Note: It is referenced from catalog.xml.]
- threddsConfig.xml - configuration file for allowing non-default services, configuring caching, etc (more details available here).
- logs/
  - catalogInit.log - log file for messages generated while reading TDS configuration catalogs during TDS initialization and reinitialization.
  - threddsServlet.log - log messages about individual TDS requests, including any error messages. Useful for debugging problems.
- cache/ - various cache directories
  - agg/
  - cdm/
  - collection/
  - ehcache/
  - ncss/
  - wcs/

TDS Root Catalog

The main TDS configuration catalog is at ${tomcat_home}/content/thredds/catalog.xml. We ship a simple test catalog:

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="THREDDS Server Default Catalog : You must change this to fit your server!"
        xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
        xmlns:xlink="http://www.w3.org/1999/xlink">

  <service name="all" base="" serviceType="compound">
    <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
    <!--service name="wms" serviceType="WMS" base="/thredds/wms/" /-->
    <!--service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" /-->
  </service>

  <datasetRoot path="test" location="content/testdata/" />

  <dataset name="Test Single Dataset" ID="testDataset" serviceName="odap"
           urlPath="test/testData.nc" dataType="Grid"/>

  <dataset name="Test Single Dataset 2" ID="testDataset2" serviceName="odap" urlPath="test/testData2.grib2" dataType="Grid"/>

  <datasetScan name="Test all files in a directory" ID="testDatasetScan" path="testAll" location="content/testdata">

    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>Grid</dataType>
    </metadata>

    <filter>
      <include wildcard="*eta_211.nc"/>
    </filter>

  </datasetScan>

  <catalogRef xlink:title="Test Enhanced Catalog" xlink:href="enhancedCatalog.xml" name=""/>

</catalog>

When the TDS starts, this main configuration catalog is read as are all catalogs in the catalog tree defined by catalogRef elements. The TDS gathers configuration information on what datasets it should serve from the datasetRoot and datasetScan elements.

Exercise: Add NCEP NAM model data

Modify the main TDS configuration catalog to include some model data.

Take a look at the data (note: the filenames may be different):

$ ls /data/ldm
bufr    dusk         dusk.080527  fsl     ldm.pq  ltng   mcidas    ngrid  nogaps    rcm      severe  surface   wseta
cosmic  dusk.080522  forecasts    gempak  logs    madis  nam_12km  nldn   rawfiles  rtmodel  suomi   upperair
$ ls /data/ldm/nam_12km
2008110312f000.grib2  2008110318f045.grib2  2008110406f003.grib2  2008110412f048.grib2  2008110500f006.grib2  2008110506f051.grib2
2008110312f003.grib2  2008110318f048.grib2  2008110406f006.grib2  2008110412f051.grib2  2008110500f009.grib2  2008110506f054.grib2
...
2008110318f033.grib2  2008110400f078.grib2  2008110412f036.grib2  2008110418f081.grib2  2008110506f039.grib2  2008110512f084.grib2

Edit the main TDS configuration catalog:

$ cd ${tomcat_home}/content/thredds
$ vi catalog.xml         // Use the editor of your choice

And add a datasetScan element for the NAM data:

<datasetScan name="NCEP NAM 12km" ID="NAM_12km"
             path="nam_12km" location="/data/ldm/nam_12km">

    <metadata inherited="true">
        <serviceName>odap</serviceName>
    </metadata>

</datasetScan>

Restart Tomcat so the TDS is reinitialized:

$ cd ${tomcat_home}/bin
$ ./shutdown.sh
$ ./startup.sh

Test that the new datasetScan is working:
1. Bring the catalog up in a browser: http://localhost:8080/thredds/catalog.html
2. Click down to one of the NAM dataset pages.
3. Select the OPeNDAP link

Not working?

Take a look at the catalogInit.log:

$ cd ${tomcat_home}/content/thredds/logs
$ more catalogInit.log

Take a look at the threddsServlet.log:

$ cd ${tomcat_home}/content/thredds/logs
$ more threddsServlet.log

Unidata

Basic TDS Configuration

THREDDS Catalog Review

Example Catalog

Notes

Constructing an access URL

Example Catalog - shared service name

Notes

TDS Configuration Catalogs

datasetRoot Element

datasetScan Element

TDS Configuration Directory

Location of the TDS Configuration Directory

TDS Root Catalog

Exercise: Add NCEP NAM model data

Not working?

`datasetRoot` Element

`datasetScan` Element