CDM Remote Web Service (Experimental)

CDM Remote is a web service implemented in the CDM library (client) and TDS (server), providing remote access to CDM datasets, using ncstream as the on-the-wire protocol. It provides access at the NetcdfFile and FeatureDataset levels of the CDM API, so there are two levels of services:

  1. CdmRemote provides index subsetting on remote CDM datasets, with the same functionality that NetcdfFile provides for CDM local datasets and DODSNetcdfFile provides for remote OPeNDAP datasets. CdmRemote supports the full CDM data model.
  2. CdmrFeature provides coordinate subsetting on remote CDM Feature Datasets, with similar functionality to WCS and Unidata's experimental NetCDF Subset Service (NCSS). Currently (version 4.2) it provides a remote API for Point and StationTimeSeries Feature Datasets.

The CDM Remote services and protocol are experimental and should not be used outside of the CDM stack for now.

CdmRemote Protocol (Data Access and Coordinate Systems)

The client forms requests of the form endpoint?query. The possible query parameters are:

  req=( CDL | NcML | capabilities | header | data)
  var=vars
where:
  vars := varspec | varspec[';' varspec]
  varspec := varname[subsetSpec]
  varname := escaped variable name
  subsetSpec := '(' fortran-90 arraySpec ')'

  fortran-90 arraySpec := dim | dim ',' dims
  dim := ':' | slice | start ':' end | start ':' end ':' stride
  slice := INTEGER
  start := INTEGER
  stride := INTEGER
  end := INTEGER

Example service requests are:

Request Response

http://server:8080/thredds/cdmremote/data.nc?req=CDL

dataset CDL
http://server:8080/thredds/cdmremote/data.nc?req=NcML dataset NcML
http://server:8080/thredds/cdmremote/data.nc?req=capabilities capabilities XML
http://server:8080/thredds/cdmremote/data.nc?req=header header ncstream message, contains the structural metadata
http://server:8080/thredds/cdmremote/data.nc?req=data&var=Temp(0:99:10,0:127,:);lat;lon data ncstream message

Client implementation

ucar.nc2.stream.CdmRemote is a subclass of NetcdfFile which provides index subsetting on remote CDM datasets. NetcdfDataset.openOrAcquireFile() looks for cdmremote:url prefix, and calls new CdmRemote(url) if found. The url must be an endpoint for a cdmremote service.

Capabilities

<cdmRemoteCapabilities location="http://localhost:8080/thredds/cdmremote/testAll/testdata/2004050300_eta_211.nc">
<featureDataset url="http://localhost:8080/thredds/cdmremote/testAll/testdata/2004050300_eta_211.nc" />
</cdmRemoteCapabilities>

CdmrFeature Protocol (Point Feature API)

The client forms requests of the form endpoint?query. The possible query parameters are:

  req=( capabilities | data | form | stations)
  accept= (csv | xml | ncstream | netcdf )
  time_start,time_end=time range
  north,south,east,west=bounding box
  var=vars
  stn=stns

where:
  vars := varName | varName[,varName]
  stns := stnName | stnName[,stnName]
  varName := valid variable name
  stnName := valid station name

PointDataset

Possible requests are:

  req=( capabilities | data | form)

Example service requests are:

Request Response Meaning
http://server:8080/thredds/cdmrfeature/data.nc/point?req=form HTML form form interface
http://server:8080/thredds/cdmrfeature/data.nc/point?req=capabilities capabilities XML describe dataset
http://server:8080/thredds/cdmrfeature/data.nc/point?req=data pointFeatureList message get all the data
http://server:8080/thredds/cdmrfeature/data.nc/point?req=data&north=40.3&south=22.8&east=-80&west=-105 pointFeatureList message get data in bounding box
http://server:8080/thredds/cdmrfeature/data.nc/point?req=data&time_start=&time_end= pointFeatureList message get data in time range
http://server:8080/thredds/cdmfeature/data.nc/point?req=data&var=Temp,lat,lon pointFeatureList message get data for listed variables

StationTimeSeries

Possible query parameters are:

  req=( capabilities | data | form | stations)

Example service requests are:

Request Response Meaning
http://server:8080/thredds/cdmrfeature/data.nc/station?req=form HTML form form interface
http://server:8080/thredds/cdmrfeature/data.nc/station?req=capabilities capabilities XML describe dataset
http://server:8080/thredds/cdmrfeature/data.nc/station?req=stations stationListMessage get all stations
http://server:8080/thredds/cdmrfeature/data.nc/station?req=stations&north=40.3&south=22.8&east=-80&west=-105 stationListMessage get stations in bounding box
http://server:8080/thredds/cdmrfeature/data.nc/station?req=stations&stn=KDEN,KLOG,MOAS stationListMessage get stations in list
http://server:8080/thredds/cdmrfeature/data.nc/station?req=data&time_start=&time_end=&stn=KDEN pointFeatureList message get data in time range
http://server:8080/thredds/cdmrfeature/data.nc/station?req=data&stn=KDEN,KLOG,MOAS pointFeatureList message get data for station list
http://server:8080/thredds/cdmrfeature/data.nc/station?req=data&north=40.3&south=22.8&east=-80&west=-105 pointFeatureList message get data in bounding box
http://server:8080/thredds/cdmrfeature/data.nc/station?req=data&var=Temp,lat,lon&stn=KDEN pointFeatureList message get data for listed variables

Questions:

Client implementation

ucar.nc2.stream.CdmRemoteFeatureDataset.factory() makes an HTTP request to endpoint+"?req=capabilities" to obtain the GetCapabilities XML document. This document describes what feature type(s) the dataset supports, which determines what are the valid requests, along with the endpoints for those feature types, which may be different than the original endpoint.

FeatureDatasetFactoryManager.open() looks for cdmremote:url prefix, and calls CdmRemoteFeatureDataset.factory(url) if found. ThreddsDataFactory.openFeatureDataset() looks for catalog InvDatasets with featureType.isPointFeatureType() and ServiceType.CdmRemote, and if found, calls CdmRemoteFeatureDataset.factory().

GetCapabilities XML document

This is an ad-hoc format (for now), example:

<?xml version="1.0" encoding="UTF-8"?>
<capabilities location="http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome">
  <featureDataset type="station" url="http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome/station" />
<variable name="record.air_pressure_at_sea_level" type="float">
<attribute name="long_name" value="Air pressure at sea level" />
<attribute name="standard_name" value="air_pressure_at_sea_level" />
<attribute name="_FillValue" type="float" value="-99999.0" />
<attribute name="units" value="hectoPascal" />
</variable>
<variable name="record.air_temperature" type="float">
<attribute name="long_name" value="Air temperature at 2 meters" />
<attribute name="standard_name" value="air_temperature" />
<attribute name="_FillValue" type="float" value="-99999.0" />
<attribute name="units" value="Celsius" />
</variable>
...
<LatLonBox>
<west>-180.0000</west>
<east>180.0000</east>
<south>-90.0000</south>
<north>82.5199</north>
</LatLonBox>
<AcceptList>
<accept>raw</accept>
<accept>xml</accept>
<accept>csv</accept>
<accept>netcdf</accept>
<accept>ncstream</accept>
</AcceptList>
</capabilities>

 


PointStream Grammer

An pointstream is a sequence of one or more messages:

   pointstream = {message}
   message = stationListMessage | pointFeatureListMessage | errorMessage | endMessage
   stationListMessage = MAGIC_StationList, vlen, PointStreamProto.StationList 
   pointFeatureListMessage = pointFeatureCollectionMessage, {pointFeatureMessage}, (endMessage | errorMessage)


   pointFeatureCollectionMessage = MAGIC_PointFeatureCollection, vlen, PointStreamProto.PointFeatureCollection
   pointFeatureMessage = MAGIC_PointFeature, vlen, PointStreamProto.PointFeature
   endMessage = MAGIC_END
   errorMessage = MAGIC_ERR, vlen, NcStreamProto.Error
   
   vlen = variable length encoded positive integer == length of the following object in bytes


   // 8 byte constants

   MAGIC_StationList            =  0fe, 0xec, 0xce, 0xda 
   MAGIC_PointFeatureCollection =  0xfi, 0xec, 0xce, 0xba 
MAGIC_PointFeature = 0xab, 0xec, 0xce, 0xba
MAGIC_END = 0xed, 0xed, 0xde, 0xde
MAGIC_ERR = 0xab, 0xad, 0xba, 0xda

The protobuf messages are defined by


Current notes on motherlode dev

Nov-11-2010

1) web.xml has a org.springframework.web.servlet.DispatcherServlet cdmRemote with a mapping:

 <servlet-mapping>
   <servlet-name>cdmRemote</servlet-name>
   <url-pattern>/cdmremote/*</url-pattern>
 </servlet-mapping>

2) Temporarily, we have the configuration of the datasets in thredds\tds\src\main\webapp\WEB-INF\cdmRemote-servlet.xml: which configures the collectionController. We will replacce this with pure catalog configuration in 4.2+. This is now being shown in serverStartup.log

 <bean id="collectionManager" class="thredds.server.cdmremote.CollectionManager">
 <property name="collections">
  <list>
   <ref bean="ncMetars" />
   <ref bean="gempakMetars" />
   <ref bean="quickScatWinds" />
   <ref bean="ncMetarsLocal" />
   <ref bean="ncMetarsLocalHome" />
   <ref bean="gempakMetarsLocal" />
   <ref bean="gempakMetarsLocalHome" />
  </list>
 </property>
</bean>
 <bean id="ncMetars" class="thredds.server.cdmremote.CollectionBean">
   <property name="path" value="/idd/metar/ncdecoded"/>
   <property name="spec" value="/data/ldm/pub/decoded/netcdf/surface/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc"/>
   <property name="recheck" value="15 min"/>
   <property name="featureType" value="STATION"/>
   <property name="raw" value="report"/>
   <property name="resolution" value="20 min"/>
 </bean>

etc

This is now being shown in serverStartup.log:

serverStartup: CdmRemoteController collections
   /idd/metar/gempak == /data/ldm/gempak/surface/#yyyyMMdd#_sao.gem
   /idd/bufr/quickScat == /data/ldm/gempak/surface/**/#yyyyMMdd#.gem
   /idd/metar/ncdecodedLocalHome == C:/data/datasets/metars/Surface_METAR_#yyyyMMdd_HHmm#.nc
   /idd/metar/ncdecoded == /data/ldm/pub/decoded/netcdf/surface/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc
   /idd/metar/gempakLocalHome == C:/data/formats/gempak/surface/#yyyyMMdd#_sao.gem
   /idd/metar/gempakLocal == D:/formats/gempak/surface2/#yyyyMMdd#_sao.gem
   /idd/metar/ncdecodedLocal == Q:/station/ldm/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc

So the URLs are

 http://server:port/thredds/cdmremote/path?req
eg:
 http://motherlode.ucar.edu:8081/thredds/cdmremote/idd/metar/gempak
 http://motherlode.ucar.edu:8081/thredds/cdmremote/idd/metar/ncdecoded
 http://motherlode.ucar.edu:8081/thredds/cdmremote/idd/bufr/quickScat
 http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocal
 http://localhost:8080/thredds/cdmremote/idd/metar/ncdecodedLocalHome?req=form

3) The default output is the dataset description: should be getCapabilities ????

FeatureDataset on location= http://localhost:8080/thredds/cdmremote/local/metars/collection
  featureType= STATION
  title= null
  desc= null
  range= start= 2006-03-25 00:00:00Z end= 2006-09-25 00:00:00Z duration= 6.0453 months resolution= null
  start= 2006-03-25 00:00:00Z
  end  = 2006-09-25 00:00:00Z
  bb   =  ll: 90.0S 180.0W+ ur: 82.51N 180.0E
  bb   =  lat= [-90.00,82.51] lon= [-180.00,180.00]
  has netcdf = true
  Data Variables (0)

parseInfo=


FeatureCollection 0 
 STATION Q:/station/ldm/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc
   npts = -1 
     bb =  lat= [-90.00,82.51] lon= [-180.00,180.00] 


4) In ToolsUI, FeatureType/PointFeature Tab:

cdmremote:http://localhost:8080/thredds/cdmremote/local/metars/collection

 
FeatureDataset on location= cdmremote:http://localhost:8080/thredds/cdmremote/local/metars/collection
  featureType= STATION
  title= METAR Data from NWS
  desc= Metar Data from NWS distributed through the Unidata IDD
    realtime datastream. 1 day's worth of data
  range= null
  start= Unknown
  end  = Unknown
  bb   = null
  has netcdf = true
  Attributes
    title = "METAR Data from NWS"
    version = 2.3
    processor = "metar2nc  version v1.2"
    Conventions = "Unidata Observation Dataset v1.0"
    standard_name_vocabulary = "CF-1.0"
    description = "Metar Data from NWS distributed through the Unidata IDD\n    realtime datastream. 1 day\'s worth of data"
    time_coordinate = "time_observation"
    cdm_datatype = "Station"
    stationDimension = "station"
    station_id = "station_id"
    station_description = "station_description"
    latitude_coordinate = "latitude"
    longitude_coordinate = "longitude"
    altitude_coordinate = "altitude"
    geospatial_lat_max = "90.0"
    geospatial_lat_min = "-90.0"
    geospatial_lon_max = "360.0"
    geospatial_lon_min = "0.0"
    time_coverage_start = "1143243900"
    time_coverage_end = "1143330240"
    observationDimension = "recNum"
  Data Variables (0)

parseInfo=


FeatureCollection 0 
 STATION cdmremote:http://localhost:8080/thredds/cdmremote/local/metars/collection
   npts = -1 
     bb =  lat= [-90.00,82.51] lon= [-180.00,180.00] 

Ask for station data, on the server, its rather slow:

CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060325_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060326_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060327_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060328_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060329_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060330_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060331_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060401_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060402_0000.nc
CompositeStationFeatureIterator open datasetQ:\station\ldm/metar/Surface_METAR_20060629_0000.nc
 sent 481 features to 20V 
 

  (check to see what the strategy is for stations: coomplete scan or ??)

5) cdmremoteCatalog.xml defines datasets for the collections we have working so far.



This document is maintained by John Caron and was last updated April 2015