NetCDF Subset Service

Please note that the interface described here is still a prototype, and subject to change.

Please send comments to the THREDDS email group (preferred) or to John Caron

Overview

The NetCDF Subset Service is a web service for subsetting CDM scientific datasets. The subsetting is specified using earth coordinates, such as lat/lon bounding boxes and date ranges, rather than index ranges that refer to the underlying data arrays. The data arrays are subsetted but not resampled or reprojected, and preserve the resolution and accuracy of the original dataset.

A Dataset is described by a Dataset Description XML document, which describes the dataset in enough detail to enable a programmatic client to form valid data requests.

The NetCDF Subset Service may return netCDF binary files (using CF-1.0 when possible), XML, or ASCII, depending on the request and the dataset.

The NetCDF Subset Service uses HTTP GET with key-value pairs (KVP). The service interface tries to follow REST design, as well as Google/KML and W3C XML Schema Datatypes when applicable.

Currently both Grids and Station data can be used with this service. See:

Subsetting Parameters (summary)

A. Specify variables

var=name of variables, separated by ',' (comma).

The list of valid variables is available from the Dataset Description.

Examples:

var=QCSurface
var=QC,LZT,PQ
var=T&var=Pressure&var=RH (allowed, but comma seperated names are preferred)

Variable names with spaces or other illegal characters must be escaped.

B. Specify spatial extent

Latitude, longitude values are specified in decimal degrees north and east, respectively.

1. Specify lat/lon bounding box

Specify all of these parameters (order does not matter):

north: latitude north (decimal degrees)
south: latitude south (decimal degrees)
east: longitude east (decimal degrees)
west: longitude west (decimal degrees)

The bounding box has west as its west edge, includes all points going east until the east edge. Units must be degrees east, may be positive or negative, and will be taken modulo 360. Therefore, when crossing the dateline, the west edge may be greater than the east edge. Examples:

north=17.3&south=12.088&west=140.2&east=160.0

2. Specify lat/lon point

latitude: latitude of point, decimal degrees north
longitude: longitude of point, decimal degrees east

The requested point must lie within the dataset spatial range. For observations, the station closest to the requested point will be used. For grids, the grid cell containing the requested point will be used.

Examples:

latitude=17.3&longitude=140.2

3. Specify station list

stn=name of stations, separated by ',' (comma)

This can only be used for station datasets. The list of valid stations is available from the Dataset Description. Station names with spaces or other illegal characters must be escaped.

Examples:

stn=KDEN
stn=KDEN,KPAL,SDOL
stn=KDEN&stn=KPAL&stn=SDOL

4. Specify horizontal stride (Grid Subsetting only)

You can optionally take only every nth point (both the x and y dimensions).

Example:

horizStride=3 (take every 3rd point in both x and y)

C. Specify time

Use one of the following methods:

1. Time range

Specify 2 of these 3 parameters (order does not matter):

time_start: starting time as an W3Cate string or "present"
time_end: ending time as an W3C date string or "present"
time_duration: length of time as an W3C time duration

The intersection of the requested time range with the dataset time range will be returned.

Examples:

time_start=2007-03-29T12:00:00Z&time_end=2007-03-29T13:00:00Z (between 12 and 1 pm Greenwich time)
time_start=present&time_duration=P3D (get 3 day forecast starting from the present)
time_end=present&time_duration=PT3H (get last 3 hours)

2. Time point

time: time as an W3C date string or "present"

The requested time point must lie within the dataset time range. The time slice/point closest to the requested time will be returned.

Examples:

time=2007-03-29T12:00:00Z
time=present

D. Specify the return format

Specify the return format(s) that you want by using the accept parameter:

accept=mime_type[,mime-type][,mime-type]

The list of possible return formats varies depending on the dataset, and can be found in the Dataset Description Document. Your request specifies the list of acceptable types, if none are valid a 400 "Bad Request" HTTP status is returned. If you specify multiple mime-types, the server will choose one of them.

The server returns the actual return format in the Content-Type header, examples:

Content-Type: text/plain;charset=ISO-8859-1
Content-Type: application/x-netcdf

Query examples:

accept=application/x-netcdf requests a netcdf file
accept=netcdf requests a netcdf file
accept=ascii,csv requests either an ascii or csv document back

The possible mime-types and aliases:

Mime Type	Synonyms
text/plain	raw, ascii
application/xml	xml
text/csv	csv
text/html	html
application/x-netcdf	netcdf

The list of actual return formats depends on the dataset, and can be found in the Dataset Description Document.

E. Specify the vertical coordinate

For Grid as Point only , you may specify the vertical coordinate. The vertical slice closest to the requested coordinate will be returned. The possible vertical coordinates are shown in the Dataset Descriptor document. If you specify multiple variables with different vertical coordinates, ambiguous results are possible, and you are advised not to use the service in this way. If you do, the following rules are used:

Variables are examined in order they appear in the query, and the vertical coordinate for the first variable that has one is used.
The service loops through the levels of that vertical coordinate, and for each variable looks for the closest value (ignoring units!!)
If the value is outside the range for that variable, a missing value (NaN) is used.

Example:

vertCoord=850

F. Addding Lat/Lon arrays to the file

If the grid is a lat/lon grid, the lat and lon coordinates will be automatically included (as 1D coordinate variables). When the grid is on a projection, the lat/lon information will not be included unless the query parameter addLatLon is present. In that case, the lat, lon coordinates will be calculated and included into the file (as 2D variables).

The 4 corners of the lat/lon bounding box are converted into projection coordinates, then the smallest rectangle including those 4 points is used.

Dataset Descriptions

Each dataset has an XML document called the Dataset Description Document. These are intended to perform the same function as OGC GetCapabilities or Atom Introspection, that is, provide clients with the necessary information to formulate a valid request and send it to the server. The content of these documents is still evolving.

Station Observation Dataset

A Station Observation Dataset is a collection of time series of observations at named locations called stations.

The dataset is described by a stationObsDataset document, which in turn points to the list of valid stations in a seperate stationCollection document. The stationCollection document can be quite large, and caching on the client (eg using the If-Modified-Since header) is an important optimization.

Example stationObsDataset document (offline example)
Example stationCollection document (offline example)

Grid Dataset

A Grid Dataset is a collection of Grids which have horizontal (x,y) coordinates, and optional vertical and time coordinates. Grid data points next to each other in index space are next to each other in coordinate space.

Example gridDataset document (offline example)

Reference

W3C Time Duration

The lexical representation for duration is the[ISO 8601] extended format PnYn MnDTnH nMnS, where nY represents the number of years, nM the number of months, nD the number of days, 'T' is the date/time separator, nH the number of hours, nM the number of minutes and nS the number of seconds. The number of seconds can include decimal digits to arbitrary precision.

The values of the Year, Month, Day, Hour and Minutes components are not restricted but allow an arbitrary unsigned integer, i.e., an integer that conforms to the pattern [0-9]+.. Similarly, the value of the Seconds component allows an arbitrary unsigned decimal. Following [ISO 8601], at least one digit must follow the decimal point if it appears. That is, the value of the Seconds component must conform to the pattern [0-9]+(\.[0-9]+)?. Thus, the lexical representation of duration does not follow the alternative format of § 5.5.3.2.1 of [ISO 8601].

An optional preceding minus sign ('-') is allowed, to indicate a negative duration. If the sign is omitted a positive duration is indicated. See also ISO 8601 Date and Time Formats (§D).

For example, to indicate a duration of 1 year, 2 months, 3 days, 10 hours, and 30 minutes, one would write: P1Y2M3DT10H30M. One could also indicate a duration of minus 120 days as: -P120D.

Reduced precision and truncated representations of this format are allowed provided they conform to the following:

If the number of years, months, days, hours, minutes, or seconds in any expression equals zero, the number and its corresponding designator ·may· be omitted. However, at least one number and its designator ·must· be present.
The seconds part ·may· have a decimal fraction.
The designator 'T' must be absent if and only if all of the time items are absent. The designator 'P' must always be present.

For example, P1347Y, P1347M and P1Y2MT2H are all allowed; P0Y1347M and P0Y1347M0D are allowed. P-1347M is not allowed although -P1347M is allowed. P1Y2MT is not allowed.

See XML Schema duration for full details.

W3C Dates

For our purposes, and ISO Date can be a dateTime or a date:

A dateTime has the form: '-'? yyyy '-' mm '-' dd 'T' hh ':' mm ':' ss ('.' s+)? (zzzzzz)?

where

'-'? yyyy is a four-or-more digit optionally negative-signed numeral that represents the year; if more than four digits, leading zeros are prohibited, and '0000' is prohibited (see the Note above (§3.2.7); also note that a plus sign is not permitted);
the remaining '-'s are separators between parts of the date portion;
the first mm is a two-digit numeral that represents the month;
dd is a two-digit numeral that represents the day;
'T' is a separator indicating that time-of-day follows;
hh is a two-digit numeral that represents the hour; '24' is permitted if the minutes and seconds represented are zero, and the dateTime value so represented is the first instant of the following day (the hour property of a dateTime object in the ·value space· cannot have a value greater than 23);
':' is a separator between parts of the time-of-day portion;
the second mm is a two-digit numeral that represents the minute;
ss is a two-integer-digit numeral that represents the whole seconds;
'.' s+ (if present) represents the fractional seconds;
zzzzzz (if present) represents the time zone (as described below).

For example, 2002-10-10T12:00:00-05:00 (noon on 10 October 2002, Central Daylight Savings Time as well as Eastern Standard Time in the U.S.) is 2002-10-10T17:00:00Z, five hours later than 2002-10-10T12:00:00Z.

A date is the same as a dateTime without the time part : '-'? yyyy '-' mm '-' dd zzzzzz?

See XML Schema dateTime and date for full details

REST Design

1. What are the resources/URIs?

The resources are THREDDS datasets. The resource URIs can be discovered in a THREDDS catalog, by looking for datasets that use the NetcdfSubset Service type. Generally these resource URLs look like:

http://servername:8080/thredds/ncss/{path/dataset}

http://servername:8080/thredds/ncss/grid/{path/dataset}

Typically the user wants a subset of the dataset.This is considered a view of a resource, rather than a separate resource:

http://servername:8080/thredds/ncss/{path/dataset}?{subset}

A desired representation of the resource is specified using the accept parameter. Again, different representations are not considered separate resources. Following the Accept HTTP header, accept takes a comma delimited list of mime-types (or aliases), but does not allow wild cards (*) or q parameters.

http://servername:8080/thredds/ncss/{path/dataset}?{subset}&accept={mime-type}

2. What's the format/representation?

The dataset itself has two representations:

an XML file describing the dataset, called the Dataset Description
- http://servername:8080/thredds/ncss/{path/dataset}/dataset.xml
An HTML form to allow interactive user input:

http://servername:8080/thredds/ncss/{path/dataset}/dataset.html

Results of a subset request can be:

a netCDF binary file
- http://servername:8080/thredds/ncss/{path/dataset}?{subset}&accept=application/x-netcdf
an XML document
- http://servername:8080/thredds/ncss/{path/dataset}?{subset}&accept=application/xml
ASCII text
- http://servername:8080/thredds/ncss/{path/dataset}?{subset}&accept=text/plain
Excel CSV (comma separated values)
- http://servername:8080/thredds/ncss/{path/dataset}?{subset}&accept=text/csv

The netCDF binary file will be encoded using CF conventions when possible, and when not possible, the encoding will be submitted to CF for approval.
The XML, ASCII, and CSV files are intended for use only for small extractions of data, and are generally missing some or all of the metadata of the dataset.
Multiple accept values can be specified, eg accept=xml,csv (comma delimited, no spaces). The server will select from that list.

Representation types

Mime Type	Synonyms
text/plain	raw, ascii
application/xml	xml
text/csv	csv
text/html	html
application/x-netcdf	netcdf

3. What are the Methods?

Only the GET method is allowed.

4. What Status codes can be returned?

200 : success
307 : redirect to an authorization challenge
400 : malformed request
401 : authorization challenge
404 : unknown resource
501 : request requires too much server resources / data result would be too large

REST Resources:

Implementing REST Web Services: Best Practices and Guidelines

NetCDF Subset Service Reference

Please note that the interface described here is still a prototype, and subject to change.

Contents:

Overview

Subsetting Parameters (summary)

A. Specify variables

B. Specify spatial extent

1. Specify lat/lon bounding box

2. Specify lat/lon point

C. Specify time

1. Time range

2. Time point

D. Specify the return format

E. Specify the vertical coordinate

F. Addding Lat/Lon arrays to the file

Dataset Descriptions

Station Observation Dataset

Grid Dataset

Reference

W3C Time Duration

W3C Dates

REST Design

1. What are the resources/URIs?

2. What's the format/representation?

3. What are the Methods?

4. What Status codes can be returned?

REST Resources: