Using NcML in TDS

An NcML document is an XML document that uses the NetCDF Markup Language to define a NetCDF dataset. NcML can be embedded directly into the TDS catalogs to achieve a number of powerful features, shown below. This embedded NcML is only useful in the TDS server catalogs, it is not meaningful to a THREDDS client, and so is not included in the client catalogs.

One can put an NcML element inside a dataset element, in which case it is a self-contained NcML dataset, or inside a datasetScan element, where it modifies a regular dataset. In both cases, we call the result a virtual dataset, and you cannot serve a virtual dataset with a file-serving protocol like FTP or HTTP. However, you can use subsetting services like OPeNDAP, WCS and NetcdfSubset.

Using NcML in a dataset element

NcML embedded in a TDS dataset element creates a self-contained NcML dataset. The TDS dataset does not refer to a data root, because the NcML contains its own location. The TDS dataset must have a unique URL path (this is true for all TDS datasets), but unlike a regular dataset, does not have to match a data root.

Modifying an existing dataset

You can use use NcML to modify an existing CDM dataset (see the NcML docs for all your options):

<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         name="THREDDS-IDD OPeNDAP Data Server" version="1.0.1">

(1)<service name="all" serviceType="Compound" base="">     
     <service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/"/>
     <service name="WCSServer" serviceType="WCS" base="/thredds/wcs/" />
   </service>

(2)<dataset name="Example NcML Modified" ID="ExampleNcML-Modified" urlPath="ExampleNcML/Modified.nc">
     <serviceName>all</serviceName>

(3)  <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/data/nc/example.nc">
(4)    <variable name="Temperature" orgName="T"/>
(5)    <variable name="ReletiveHumidity" orgName="rh">
(6)      <attribute name="long_name" value="relatively humid"/>
         <attribute name="units" value="percent (%)"/>
(7)      <remove type="attribute" name="description"/>
       </variable >
     </netcdf>

  </dataset>
</catalog>

A compound service is defined that allows the virtual dataset to be served through both OPENDAP and WCS. Make sure that the base attributes are exactly as shown
The dataset is given a urlPath of "ExampleNcML/Modified.nc". The urlPath is essentially arbitrary, but must be unique within the TDS, and you should maintain a consistent naming convention to ensure uniqueness, especially for large collections of data. Its important to also give the dataset a unique ID.
An NcML dataset is defined which references the netCDF file at the absolute location /data/nc/example.nc. Note that you must declare the NcML namespace exactly as shown.
The variable named T in the original file is renamed Temperature.
The variable named rh in the original file is renamed ReletiveHumidity.
Two attributes of rh are defined, long_name and units. If these already exist, they are replaced.
The attribute of rh called description is removed.

Using NcML aggregation

Here is an example that defines a dataset using NcML aggregation.

<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         name="THREDDS-IDD OPeNDAP Data Server" version="1.0.1">

(1)<service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/" />

(2)<dataset name="WEST-CONUS_4km Aggregation" ID="SSEC/IDD-Satellite/3.9/WEST-CONUS_4km-Agg" urlPath="satellite/3.9/WEST-CONUS_4km">
(3)  <serviceName>ncdods</serviceName>

(4)  <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
(5)   <aggregation dimName="time" type="joinNew">
(6)     <variableAgg name="IR" />
(7)     <scan dateFormatMark="WEST-CONUS_4km_3.9_#yyyyMMdd_HHmm" location="/data/ldm/pub/native/satellite/3.9/WEST-CONUS_4km/" suffix=".gini" />
      </aggregation>
     </netcdf>

   </dataset>
</catalog>

An OPENDAP service is defined called ncdods.
A THREDDS dataset is defined, which must have a urlPath that is unique within the TDS, in this case satellite/3.9/WEST-CONUS_4km.
The dataset uses the ncdods service.
An NcML netcdf element is embedded inside the THREDDS dataset element.
An NcML aggregation of type joinNew is declared, using a new dimension called time.
The variable named IR will be aggregated.
All the files in the directory /data/ldm/pub/native/satellite/3.9/WEST-CONUS_4km/ that end with .gini will be scanned to create the aggregation.

See NcML Aggregation for more details.

Using NcML in a datasetScan element

If an NcML element is added to a DatasetScan, it will modify all of the datasets contained within the DatasetScan. It is not self-contained, however, since it gets its location from the datasets that are dynamically scanned.

(1)<datasetScan name="Ocean Satellite Data" ID="ocean/sat" path="ocean/sat" dirLocation="R:/tds/netcdf/" filter=".*\.nc$">
(2)  <metadata inherited="true">
       <serviceName>ncdods</serviceName>
       <dataType>Grid</dataType>
     </metadata>
(3)  <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
       <attribute name="Conventions" value="CF-1.0"/>
     </netcdf>
(4)  <addID />
   </datasetScan>

A datasetScan element is created whose contained datasets start with URL path ocean/sat, and whose contents are all the files in the directory R:/testdata/tds/netcdf/ which end in .nc.
All contained datasets inherit metadata indicating they use the ncdods service and are of type Grid.
All contained datasets are wrapped by this NcML element. In this case, each dataset has the global attribute :Conventions="CF-1.0" added to it. Note that there is no location attribute, which is implicitly supplied by the datasets found by the datasetScan.
This element causes ID attributes to be added to each of the datasets contained in the DatasetScan.

Cautions

scan vs datasetScan

The scan element in the NcML aggregation is similar in purpose to the datasetScan element, but be careful not to confuse the two. The datasetScan element is more powerful, and has more options for filtering etc. Its job is to create nested dataset elements inside the datasetScan, and so has various options to add information to those nested datasets. It has a generalized framework (CrawlableDataset) for crawling other things besides file directories. The scan element's job is to easily specify what files go into an NcML aggregation, and those individual files are hidden inside the aggregation dataset. It can only scan file directories. In the future, some of the capabilities of datasetScan will migrate into NcML scan.

Cant use HTTPServer

Remember that you cant use HTTPServer for NcML datasets. Use only the subsetting services OpenDAP, WCS and NetcdfSubset.

Debugging tips

When things go wrong, its best to first debug the aggregation outside of the TDS:

Go to the TDS catalog and find the problem dataset. Inside the <dataset> element will be a <netcdf> element, that is the NcML aggregation. Extract it out and put it in a file called "test.ncml".
1. Add the XML header to the top of it: <?xml version="1.0" encoding="UTF-8"?>
2. Remove the recheckEvery attribute if present on the <scan> element.
3. Make sure that the <scan> location is available on the machine you are running ToolsUI
Now start up ToolsUI, and in the viewer tab, navigate to test.ncml and try to open it.
If the dataset is dynamic ( files can be added or deleted), add the recheckEvery attribute on the scan element and open the dataset, then reopen after a new file has arrived (and recheckEvery time has passed). Generally you make recheckEvery very short while testing.
Now add the NcML dataset back to the TDS, without a recheckEvery attribute on the scan element. See if OPeNDAP access works,
Add the recheckEvery attribute (if needed) and test again.