Using NcML in TDSAn NcML document is an XML document that uses the NetCDF Markup Language to define a CDM dataset. NcML can be embedded directly into the TDS catalogs to achieve a number of powerful features, shown below. This embedded NcML is only useful in the TDS server catalogs, it is not meaningful to a THREDDS client, and so is not included in the client catalogs.
One can put an NcML element inside a dataset element, in which case it is a self-contained NcML dataset, or inside a datasetScan element, where it modifies a regular dataset. In both cases, we call the result a virtual dataset, and you cannot serve a virtual dataset with a file-serving protocol like FTP or HTTP. However, you can use subsetting services like OPeNDAP, WCS, WMS and NetcdfSubset.
NcML embedded in a TDS dataset element creates a self-contained NcML dataset. The TDS dataset does not refer to a data root, because the NcML contains its own location. The TDS dataset must have a unique URL path (this is true for all TDS datasets), but unlike a regular dataset, does not have to match a data root.
You can use use NcML to modify an existing CDM dataset:
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink"
name="TDS workshop test 1" version="1.0.2">
1)<service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/"/>
2)<dataset name="Example NcML Modified" ID="ExampleNcML-Modified" urlPath="ExampleNcML/Modified.nc">
<serviceName>ncdods</serviceName>
3) <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/data/nc/example1.nc">
4) <variable name="Temperature" orgName="T"/>
5) <variable name="ReletiveHumidity" orgName="rh">
6) <attribute name="long_name" value="relatively humid"/>
<attribute name="units" value="percent (%)"/>
7) <remove type="attribute" name="description"/>
</variable >
</netcdf>
</dataset>
</catalog>
See NcML Tutorial for more details.
Lets look at serving a file directly vs serving it through NcML:
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink"
name="TDS workshop test 2" version="1.0.2">
<service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/"/>
1)<datasetRoot path="test/ExampleNcML" location="/data/nc/" />
2)<dataset name="Example Dataset" ID="Example" urlPath="test/ExampleNcML/example1.nc">
<serviceName>ncdods</serviceName>
</dataset>
3)<dataset name="Example NcML Modified" ID="Modified" urlPath="ExampleNcML/Modified.nc">
<serviceName>ncdods</serviceName>
4) <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/data/nc/example1.nc">
<variable name="Temperature" orgName="T"/>
</netcdf>
</dataset>
</catalog>
Here is an example that defines a dataset using NcML aggregation.
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink"
name="TDS workshop test 3" version="1.0.2">
1)<service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/" />
2)<dataset name="Example NcML Agg" ID="ExampleNcML-Agg" urlPath="ExampleNcML/Agg.nc">
3) <serviceName>ncdods</serviceName>
4) <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
5) <aggregation dimName="time" type="joinExisting">
6) <scan location="/data/workshop/test/cg/" dateFormatMark="CG#yyyyDDD_HHmmss" suffix=".nc" subdirs="false"/>
</aggregation>
</netcdf>
</dataset>
</catalog>
See NcML Aggregation for more details.
If an NcML element is added to a DatasetScan, it will modify all of the datasets contained within the DatasetScan. It is not self-contained, however, since it gets its location from the datasets that are dynamically scanned.
(1)<datasetScan name="Ocean Satellite Data" ID="ocean/sat" path="ocean/sat" location="R:/tds/netcdf/">
<filter>
<include wildcard="*.nc" />
</filter>
(2) <metadata inherited="true">
<serviceName>ncdods</serviceName>
<dataType>Grid</dataType>
</metadata>
(3) <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
<attribute name="Conventions" value="CF-1.0"/>
</netcdf>
</datasetScan>
The scan element in the NcML aggregation is similar in purpose to the datasetScan element, but be careful not to confuse the two. The datasetScan element is more powerful, and has more options for filtering etc. Its job is to create nested dataset elements inside the datasetScan, and so has various options to add information to those nested datasets. It has a generalized framework (CrawlableDataset) for crawling other things besides file directories. The scan element's job is to easily specify what files go into an NcML aggregation, and those individual files are hidden inside the aggregation dataset. It can only scan file directories. In the future, some of the capabilities of datasetScan will migrate into NcML scan.
Lets look at using a DatasetScan and an Aggregation scan on the same collection of files. Download catalogScan.xml, place it in your TDS content/thredds directory and add a catalogRef to it from your main catalog.
<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink"
name="TDS workshop test 4" version="1.0.2">
<service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/"/>
1) <dataset name="Example NcML Agg" ID="ExampleNcML-Agg" urlPath="ExampleNcML/Agg.nc">
<serviceName>ncdods</serviceName>
2) <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
<aggregation dimName="time" type="joinExisting" recheckEvery="4 sec">
<scan location="/workshop/test/cg/" dateFormatMark="CG#yyyyDDD_HHmmss" suffix=".nc" subdirs="false"/>
</aggregation>
</netcdf>
</dataset>
3) <datasetScan name="CG Data" ID="cg/files" path="cg/files" location="/workshop/test/cg/">
<metadata inherited="true">
<serviceName>ncdods</serviceName>
<dataType>Grid</dataType>
</metadata>
<filter>
4) <include wildcard="*.nc"/>
</filter>
5) <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
<attribute name="Yoyo" value="Ma"/>
</netcdf>
</datasetScan>
</catalog>
Start and restart your TDS and look at those datasets through the HTML interface and through ToolsUI.
A Forecast Model Run Collection is a collection of forecast model output. A special kind of aggregation, called an FMRC Aggregation, creates a dataset that has two time coordinates, called the run time and the forecast time. This dataset can then be sliced up in various ways to create 1D time views of the model output. A special kind of dataset, called a datasetFmrc makes these 1D datasets available though the TDS.
Download fmrc.ncml. and gfs.zip, unzip the latter and place in your data directory). In the ToolsUI NcML Tab, open fmrc.ncml and change the scan location to point to your data directory:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
<aggregation dimName="run" type="forecastModelRunCollection" >
<scan location="/data/working/fmrc/" suffix=".grib1"
dateFormatMark="GFS_Puerto_Rico_191km_#yyyyMMdd_HHmm" />
</aggregation>
</netcdf>
Now download catalogFmrc.xml, place it in your TDS content/thredds directory and add a catalogRef to it from your main catalog:
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink"
name="THREDDS Data Server" version="1.0.1">
1)<service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/" />
2)<datasetFmrc name="GFS_Puerto_Rico_191km" path="fmrc/NCEP/GFS/Puerto_Rico_191km" >
3) <metadata inherited="true">
<serviceName>ncdods</serviceName>
<dataFormat>GRIB-1</dataFormat>
<documentation type="summary">Specially good GFS_Puerto_Rico_191km</documentation>
</metadata>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
4) <aggregation dimName="run" type="forecastModelRunCollection">
5) <scan location="/workshop/test/gfs/" suffix=".grib1"
6) dateFormatMark="GFS_Puerto_Rico_191km_#yyyyMMdd_HHmm"/>
</aggregation>
</netcdf>
</datasetFmrc>
</catalog>
The contained datasets include the resulting 2D time dimension dataset, as well as the 1D time views of the ucar.nc2.dt.fmrc.ForecastModelRunCollection dataset described in section 4 above, as seen in the resulting HTML page for that dataset:

Also see: FMRC poster
When things go wrong, its best to first debug the aggregation outside of the TDS:
Remember that you can't use HTTPServer for NcML datasets. Use only the subsetting services OpenDAP, WCS, WMS, and NetcdfSubset.
This document is maintained by John Caron and was last updated July 2009