•         advanced

NcML Tutorial

Introduction

A netCDF-3 file is a physical file that uses the netCDF file format (version 1). A netCDF-4 file is a physical file that uses a specialized version of the HDF-5 file format, called the netCDF-4 file format. The NetCDF-Java library can read a number of other binary file formats, such as GRIB and NEXRAD radar files, and make the data accessible through the standard netCDF API. A netCDF dataset, then, is any scientific data collection which is accessible through the netCDF API.

The NetCDF Markup Language (NcML) is an XML dialect that allows you to create netCDF datasets. An NcML document is an XML document that uses NcML, and defines a netCDF dataset. Commonly, the NcML document refers to another netCDF dataset called the referenced netCDF dataset. The purpose of NcML is to allow:

1. Metadata to be added, deleted, changed and in general redefined.
2. Variables to be renamed, added, deleted and restructured.
3. Data from multiple NetCDF files to be combined.

See also: Annotated NcML Schema

1. Completely specify the NetCDF Dataset Metadata

In this example, the use of the <explicit/> element means that all of the metadata is specified in the XML, and the referenced dataset is used only for data access. If there are any other attributes, variables or dimensions in the referenced dataset, they are ignored.

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" 
     location="file:/dev/netcdf-java-2.2/test/data/example1.nc">

  <explicit/>

  <dimension name="time" length="2" isUnlimited="true" />
  <dimension name="lat" length="3" />
  <dimension name="lon" length="4" />

  <attribute name="title" type="String" value="Example Data" />

  <variable name="rh" shape="time lat lon" type="int">
    <attribute name="long_name" type="String" value="relative humidity" />
    <attribute name="units" type="String" value="percent" />
  </variable>

  <variable name="T" shape="time lat lon" type="double">
    <attribute name="long_name" type="String" value="surface temperature" />
    <attribute name="units" type="String" value="C" />
  </variable>

  <variable name="lat" shape="lat" type="float">
    <attribute name="units" type="String" value="degrees_north" />
    <values>41.0 40.0 39.0</values>
  </variable>

  <variable name="lon" shape="lon" type="float">
    <attribute name="units" type="String" value="degrees_east" />
    <values>-109.0 -107.0 -105.0 -103.0</values>
  </variable>

  <variable name="time" shape="time" type="int">
    <attribute name="units" type="String" value="hours" />
    <values>6 18</values>
  </variable>
</netcdf>

Data for the lat, lon and time coordinate variables comes from the NcML document, while the data for the rh and T variables will be obtained from the referenced dataset at URL file:/dev/netcdf-java-2.2/test/data/example1.nc. Note that this is a URL that uses the file: scheme. It is allowable to omit the file: prefix, but for clarity we recommend that you use it.

This creates the following NetCDF Dataset, in CDL notation:

netcdf testRead.ncml {

 dimensions:
  time = UNLIMITED;   // (2 currently)   // (has coord.var)
  lat = 3;   // (has coord.var)
  lon = 4;   // (has coord.var)

 variables:
  int rh(time, lat, lon);
   :long_name = "relative humidity";
   :units = "percent";
  double T(time, lat, lon);
   :long_name = "surface temperature";
   :units = "C";
  float lat(lat);
   :units = "degrees_north";
  float lon(lon);
   :units = "degrees_east";
  int time(time);
   :units = "hours";
 // Global Attributes:
  :title = "Example Data";
 data:
  lat = 41.0, 40.0, 39.0;
  lon = -109.0, -107.0, -105.0, -103.0;
  time = 6, 18;
}

Allowable forms of the referenced dataset location are:

• an absolute HTTP URL (eg http://server/myfile.nc)
• An absolute file URL with an absolute path (eg file:/usr/local/data/mine.nc). You can sometimes omit the file: prefix, but not always, so we recommend using the file: prefix.
• An absolute file URL with a relative path (eg file:data/mine.nc). The file will be opened reletive to the working directory.
• A relative URL resolved against the NcML location (eg subdir/mydata.nc). You must not use a file: prefix in this case.

2. Use the metadata in the existing file

In this example, all of the metadata is read in from the underlying file.

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
  location="file:/dev/NCdataset/example.nc">

  <readMetadata/>
</netcdf>

Here, the <readMetadata/> element causes all of the metadata from the netCDF file to be read in. In this case, the NcML dataset and the referenced dataset are identical.

Since <readMetadata/> is the default, you can leave it out:

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
  location="file:/dev/NCdataset/example.nc" />

You must declare the ncml namespace in the top netcdf element. You can use a namespace prefix like nc or not:

1. No prefix: xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
2. Use "nc" prefix: xmlns:nc="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
   

3. Read in metadata from existing NetCDF file and modify:

In this example, we read in all of the metadata from the underlying file, and modify it through elements in the NcML.

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" 
      location="file:C:/dev/netcdf-java-2.2/test/data/example1.nc">

(1)<attribute name="title" type="String" value="Example Data using CF" />
(2)<attribute name="Conventions" value="CF-1.0" />

(3)<variable name="ReletiveHumidity" orgName="rh">
(4)  <attribute name="standard_name" type="String" value="relative humidity" />
   </variable>

   <variable name="T">
(5)  <attribute name="standard_name" type="String" value="temperature" />
(6)  <attribute name="units" type="String" value="degreesC" />
   </variable>

(7)<variable name="deltaLat" type="double" shape="lat">
     <values>.1 .1 .01</values>
   </variable>

</netcdf>

The absence of the <explicit/> element means that all the metadata in the referenced dataset is read in. Then:

1. The global attribute named "title" is given a new value.
2. A global attribute named "Conventions" is added.
3. The variable "rh" is renamed to "ReletiveHumidity"
4. A variable attribute "standard_name" is added
5. A variable attribute "standard_name" is added
6. The variable attribute "units" is given a new value.
7. A new variable is "deltaLat" added. Since it doesnt exist in the referenced dataset, you must also define its data values.

This creates the following NetCDF Dataset.

netcdf replace.xml {
  dimensions:
    time = UNLIMITED;   // (2 currently)   // (has coord.var)
    lat = 3;   // (has coord.var)
    lon = 4;   // (has coord.var)

  variables:
    double deltaLat(lat);
    int ReletiveHumidity(time, lat, lon);
      :long_name = "relative humidity";
      :standard_name = "relative humidity";
      :units = "percent";
    double T(time, lat, lon);
      :long_name = "surface temperature";
      :standard_name = "temperature"; 
      :units = "degreesC";
    float lat(lat);
      :units = "degrees_north";
    float lon(lon);
      :units = "degrees_east";
    int time(time);
      :units = "hours";
	  
  // Global Attributes:
  :Conventions = "CF-1.0";
  :title = "Example Data using CF";

  data:
    deltaLat = 0.1, 0.1, 0.01;
}

Using ToolsUI to generate NcML

You can use the ToolsUI NcML Tab to generate NcML from a CDM file (enter a dataset URL or browse for a local file). You can save the NcML to disk .

Writing out files modified with NcML

You can also generate a NetCDF-3 file from the NcML , and the equivalent binary NetCDF file will be created corresponding to the NcML dataset, and all the original data will be copied to it. Since the binary data is taken from the original file, this is efficient for even large files.

This works as long as you are using only netCDF-3 compatible features. In practice this means no Structures or Groups. Strings are ok, and are converted to type char. The equivilent Java code to do this is:

    NetcdfDataset ncd = NcMLReader.readNcML (ncml_filename, null);
    NetcdfFile ncdnew = ucar.nc2.FileWriter.writeToFile(ncd, fileout_name, true);
    ncd.close();
    ncdnew.close();

Next: Aggregation

This document is maintained by John Caron and was last updated on Jun 03, 2008

Contact Us   •   Site Map   •   Search   •   Terms and Conditions   •   Privacy Policy   •   Participation Policy

Unidata is a member of the UCAR Community Programs, is managed by the University Corporation for Atmospheric Research, and is sponsored by the National Science Foundation. P.O. Box 3000   •   Boulder, CO 80307-3000 USA   •   Tel: 303-497-8643   •   Fax: 303-497-8690