THREDDS Technical Status

THREDDS and Unidata

The efforts of the THREDDS group focus mainly on three themes from the Unidata 2013 proposal:

Theme 2 - Advancing data services;
Theme 3 - Developing and deploying useful tools; and
Theme 5 - Providing leadership in cyberinfrastructure.

The work of the THREDDS group is comprised of two main areas: the THREDDS Data Server (TDS) and the Common Data Model (CDM) / netCDF-Java library. The data available from the IDD is a driving force on both the TDS and netCDF-Java development. The ability to read all the IDD data through the netCDF-Java library allows the TDS to serve that data and provide services on/for that data. There is also a strong tie with the IDV group because they depend to a large degree on the netCDF-java library for access to the data.

Status update 1 March 2010

TDS 4.1 and CDM 4.1 are the current stable releases. The features and enhancements that have gone into those versions as well as plans for both 4.2 versions are given below.

THREDDS Data Server (TDS)
- TDS 4.1 was promoted to our stable release in December. Includes:
  - Improved tracking of dataset collections (used in aggregation and PointObs collections) which affects performance on large data collections.
  - CdmRemote - experimental web service for point feature datasets
  - ncstream - experimental web service for remote access to CDM datasets
  - Improved handling of GRIB ensemble data (including a new GRIB index format).
  - Performance, memory, and stability improvements
  - Uses netCDF-Java/CDM 4.1
- Plans for TDS 4.2 development
  - Refactor FMRC aggregation to simplify configuration and improve performance.
  - Improved integration with ncWMS code (easier to stay up-to-date with ncWMS code).
  - Further development of Point Feature collection services including extending it to other data types.
  - Continued refactoring of TDS configuration system for ease of configuration and improved performance.
Common Data Model / netCDF-Java
1. Netcdf-Java library version 4.1 is now the stable release
  1. Point Features
    1. CF-1.5 (provisional), Suomi, FslWindProfiler, Cosmic, GempakSurface, Madis, UnidataObs, GempakSounding, NLDN, USPLN, Madis-ACARS
    2. Handle multiple profiles in the FslRaob plugin
    3. Add new functionality to StationProfileFeature: getTimes(), getProfileByDate()
  2. Coordinate Systems
    1. CEDRIC radar
    2. Orthogonal Curvilinear (GRIB2)
    3. MSGnavigation (GRIB2 from EUMETSAT)
    4. AlbersEqualAreaEllipse, LambertConformalConicEllipse
  3. IOSP
    1. H5iosp: deal with compressed, chunked compound types
    2. BUFR: use WMO (prerelease) tables version 14, many other improvements
    3. GEMPAK sounding IOSP
    4. Improve HDF-EOS processing
  4. ToolsUI
    1. New Tabs for for examining BUFR tables B and D
  5. General
    1. Add -isLargeFile option to FileWriter to generate 64-bit offset files
    2. Experimental filesystem caching
2. Netcdf-Java library version 4.2 is the experimental release
  1. FMRC refactor
    1. No longer have to specify file structure, all files are scanned as GridDatasets and FMRC is automatically created
    2. no longer need fmrcDefinition.xml files, equivilent is generated automatically
    3. UI Tabs for examining FMRC data structures
    4. experimental caching using Berkely DB Java edition
  2. Collections specification
    1. same functionality as scan, scanFmrc, and CrawlableDataset
    2. specify a collection with a single string, eg " /data/NCEP/GFS/Alaska_191km/**/GFS_Alaska_191km_#yyyyMMdd_HHmm#\.grib"
    3. specify collection from a THREDDS catalog
    4. specify collection from NcML
  3. Improve support for arbitrary 2D Coordinates, like Orthogonal Curvilinear
3. Six month plan:
  1. Release version 4.2 with refactored FMRC
  2. Complete CF point obs conventions, for CF version 1.5
  3. Complete implementation for CF point obs.
  4. Complete point collection services infrastructure
  5. (Possible) JNA interface to netcdf-4 C library for writing netcdf-4.

Status update 10/01/2009

THREDDS Data Server (TDS)

Stable release is 4.0.26
1. Mainly a performance, memory, and stability release.
2. Uses netCDF-Java/CDM 4.0
3. Added OGC WMS service [*]
4. Support for some GRIB ensemble data.
Development release 4.1. Currently alpha, will promote to stable release by end of October
1. More performance, memory, and stability improvements.
2. Uses netCDF-Java/CDM 4.1
3. CdmRemote - experimental web service for point feature datasets
4. ncstream - experimental web service for remote access to CDM datasets
5. Improved tracking of dataset collections (used in aggregation and PointObs collections).
6. thredds.filesystem uses ehcahe to cache file directory scanning
7. Improved handling of GRIB ensemble data (including a new GRIB index format).

Common Data Model / netCDF-Java
1. Netcdf-Java library version 4.1 is in beta. It requires the use of Java 1.6.
2. Point Features
  1. "Nested Table" implementation refactor
  2. CdmRemote experimental protocol for remote access to point feature datasets using cdmremote: prefix
  3. PointDatasetCollections can be defined on the fly using collection: prefix
3. Coordinate Systems
  1. AvhrrConventions
  2. NppConventions
  3. Cosmic Conventions
  4. WRF MAP_PROJ=0
  5. 2D Coordinate Systems can now efficiently find(x,y) from (i,j)
4. IOSP
  1. GEMPAK Grid data (Don Murray)
  2. GRIB IOSP deals with ensembles, new binary indexes (gbx8) with complete PDS, GDS
  3. CAMx UAM-IV (Barron Henderson)
5. General
  1. udunits: logarithmic units (Steve Emmerson)
  2. catalog classes refactored to thredds.catalog2
  3. ncstream experimental protocol for remote access to CDM datasets using ncstream: prefix
  4. mock testing framework
6. ToolsUI
  1. New Tabs for debugging GRIB and HDF4 IOSPS
  2. TDS Monitor log analysis application (experimental)
7. NcML
  1. Values can be specified with fromAttribute
  2. logicalSection, logicalSlice
  3. DatasetCollectionManager for managing aggregation and Point Feature collections

[*] Based on ncWMS code, with thanks to Jon Blower (Univ of Reading) and Pauline Mak (Univ of Tasmania).

Status update 03/20/2009

THREDDS Data Server (TDS)

Stable release is 3.17.3.1, mainly in maintenance mode.

Security enhancements.
GINI polar stereographic bug fix.
Various other bug fixes.

Development release 4.0.6; alpha release, available for testing

Main focus for 4.0 release:

Use of CDM/netCDF-Java 4.0
Improvement to performance, memory use, and stability.
Addition of OGC WMS service [based on ncWMS code, with thanks to Jon Blower (Univ of Reading) and Pauline Mak (Univ of Tasmania)]

Support for ensemble GRIB data added.
Security enhancements, from 3.17 and more.
Using Spring Framework for improved modularity and support for other industry best practices.

Common Data Model
1. Netcdf-Java library version 4.0 is now the stable release. This means that we are committed to the public API, and will keep future changes to the API backwards compatible. Much work has gone into fixing bugs and stabilizing the core code, to make it suitable in production code both as standalone library and as part of the TDS.
2. FeatureDataset
  1. FeatureDataset management extended to Grids, RadialDataset.
  2. PointDataset implementation of the "Nested Table" point data model is mostly complete, with some optimisation still to be done.
  3. Writing datasets into CF specification is started, but still needs lots of work. This will be the output format for several TDS subsetting services, including possibly a WCS point service.
3. Coordinate Systems
  1. Rotated Lat/Lon Projection
  2. add _CoordinateSystemFor to allow wholesale assignment of coord systems to variables.
  3. Modis Satellite CoordSysBuilder
4. IOSP
  1. GEMPAK Point data (Don Murray)
  2. GRIB IOSP refactored (Robb Kambic). Now uses binary indexes for 2X faster reading.
  3. HDF5 IOSP now correctly reads fractal heap structure, allows fletcher32 filter.
  4. NIDS IOSP can handle TDWR. (Yuan Ho)
  5. Universal Format Radar IOSP (Yuan Ho)
5. General
  1. FileCache refactored for additional thread safety and diagnostics.
  2. Detection of NetcdfFile object cycle mismanagement, and file handle leaks.
6. ToolsUI
  1. New Tabs for debugging BUFR and HDF5 IOSPS, Aggregated datasets, and FMRC Datasets.
  2. FeatureDataset scanner tries to guess what feature type a file is.
7. NcML
  1. Aggregation now allows elements to modify each dataset in the aggregation, before it is aggregated.
    this allows datasets to be made into grids for fmrc, for example.
  2. Rudimentary WMS client for testing TDS WMS

Status update 09/24/2008

THREDDS Data Server (TDS)
1. Stable release is 3.16.43 is mostly in maintenence mode.
  1. Level II NEXRAD radar server bug fixes
2. Development release is 4.0.24
  1. Integrating Spring into TDS for long term maintainability.
  2. New WCS 1.0, WCSplus implementations
  3. Improvements in configuration and management of TDS/IDD catalogs.
  4. Performance enhancements based on TDS profiling
  5. A prototype adding WMS functionality to TDS is done, based on Jon Blower (Readding U) ncWMS code.
Common Data Model
1. Stable release 2.2.22 is in maintenence mode, as we have run out of '2's.
2. Features backported (in both 2.2 and 4.0):
  1. McIDAS AREA file IOSP
  2. New NEXRAD hi-res file format
  3. Improved NIDS, GINI handling
3. Development release is 4.0.23
  1. BUFR IOSP is rewritten, and works on almost entire IDD/HDS BUFR messages.
  2. Enhancements like scale/offset correctly applied to structure member variables.
  3. More flexible NetcdfDataset ehancements control by user
  4. Many improvements in Structure/Sequence/ArrayStructure classes.
  5. Use of enums
  6. Starting to do object size enhancements based on TDS profiling
  7. No direct use of ucar.units for time processing. Will switch to joda or similar.
4. CDM Nested Tables
  1. General API for handling point data, will replace older ucar.dt.point classes
  2. Working on CF standard for writing to netcdf-3.
5. BUFR data format
  1. We are working with international community to solve some of the problems of decoding BUFR data in a general way.
  2. We are coordinating with British Met on their efforts to create a standard XML BUFR table format.
  3. Helping NCEP debug some encoding problems.
  4. The bufrtables email discussion group is hosted at Unidata.
  5. We will soon have a BUFR validation web service to assist providers and clients.
6. NcML
  1. General refactoring for all but FMRC
7. ToolsUI
  1. FeatureType Tab supports Nested Tables implementation, will eventually supplant Point/Station/Profile/etc tabs
  2. IOSP/BUFR tab for debugging BUFR files

Status update 03/25/2008

THREDDS Data Server (TDS)
1. Stable release is 3.16.35.
  1. Level II NEXRAD radar server allows user to subset by station, space, and time. Integrated with IDV chooser.
  2. Netcdf Subset for Grids as Points
  3. Write any GridDataset as CF compliant netCDF file
  4. Write UnidataStationObsDataset netCDF file
2. Most work is on the 4.0 branch, still experimental:.
  1. Level III NEXRAD radar server allows user to subset by station, space, time and product
  2. Experimental WCS 1.0+ server
  3. Integrating Spring into TDS for long term maintainability. We will incrementally move functionality over to it.
Common Data Model
1. Stable release 2.2.22.20 is in maintenence mode.
2. Current work is on a 4.0 release. A pre-alpha early access version is available.
  1. HDF4 IOSP is complete. Ongoing work to decode HDF-EOS coordinate systems
  2. GEMPAK Grid IOSP
  3. NMC Office Note 29/143 IOSP for CADIS datasets.
  4. RotatedPole projection
3. CDM Data Access
  1. Structure.select(): create subset consisting only of selected member variables
  2. Sequences allow iteration through 1D structures (experimental)
4. CDM Scientific Feature Types
  1. CDM Feature Types draft doc
  2. CDM Point Feature Types draft doc
  3. CF Conventions for Point Data (proposed)
  4. New pass on implementation in ucar.nc2.dt2 package (ongoing).
  5. CDM doc updated
5. NcML
  1. Tiled aggregation
  2. JoinExisting accepts timeUnitsChange
  3. Union accepts scan

Status update 10/22/2007

THREDDS Data Server (TDS): Current release is 3.16.
1. Current work is on a 4.0 release.
  1. Clean up / fix NcML aggregation.
  2. Uses netCDF-Java 4.0
  3. Upgrade WCS to version 1.1. Experiment with WCS-REST.
  4. NetCDF file writing consistently encoded in CF Conventions.
2. Taught 2-day Unidata workshop on THREDDS Data Server Administration.
3. Latest version of Live Access Server (LAS) from PMEL is now built on TDS.

Common Data Model: Current release is 2.2.22.
1. Current work is on a 4.0 release. A pre-alpha early access version is available.
  1. This will read netCDF-4 file formats, and hopefully allow complete access to HDF5, including HDF5-EOS. Writing netCDF-4 format may not be feasible, however.
  2. The entire library is extensively refactored, and the APIs are being stabilized based on the new features of the netCDF-4 file format.
  3. NcML is extensively refactored, in order to add new features (like tiled aggreagtion).
2. Preparing to work with OPeNDAP to continue data model reconciliation, as part of the SDCI proposal.
3. Taught 1-day Unidata workshop on Writing an I/O Service Provider
4. CF working group for encoding Point Observation datasets in netCDF.

International Standards:
1. Open Geospatial Consortium (OGC) fall meeing hosted by UCAR. One of the days was OGC Interoperability Day at Unidata: Standards-based Web Services Interfaces to Existing Atmospheric/Oceanographic Data Systems, where 50+ participants explored connections bewteen ISO/OGC standards and Unidata and related technologies.
2. Working with an international group (British Atmospheric Data Center, British Met, and others) to define WCS 1.0 extentions, and help drive further version of WCS.

Status update 07/02/2007

THREDDS Data Server (TDS): Current release is 3.16.
1. The NetCDF Subset Service is an experimental REST web service for subsetting CDM scientific datasets. The subsetting is specified using earth coordinates, such as lat/lon bounding boxes and date ranges, rather than index ranges that refer to the underlying data arrays. The data arrays are subsetted but not resampled or reprojected, and preserve the resolution and accuracy of the original dataset.
2. New authorization/authentication capabilities allow servers to be configured with restricted dataset access.
3. Forecast Model Run Collections : Automatically create 1D subsets of 2D time datasets. (Example)
4. Runtime configuration completed, allows users to configure, load classes at runtime.
5. Dataset Source plugin: allows users to create custom and virtual datasets.
6. Refactor of dataset filter interface (CrawlableDatasetFilter) simplifies new implementations
7. TDS Tutorial will be updated for next Unidata workshop.
8. OPeNDAP server: direct implementation of ascii ascii response (not recursive call)

Common Data Model: Current release is 2.2.20, last stable release using Java 1.4.
1. Joint development with opendap.org
  - Proposal accepted to merge OPeNDAP/netCDF library with netCDF-4 library
  - Add all necessary capabilities for HTTP authentication in OPeNDAP Java library
2. Radial Datatype Collections now can be visualizeed in the IDV
3. Starting to handle McIDAS AREA files : McIdasAreaProjections, prototype IOSP.
4. NcML
  - Specify IOSP class, parameter, buffer_size.
  - New aggregation types: ForecastModelRunCollection, ForecastModelRunSingleCollection, JoinExistingOne.
5. Runtime configuration completed, allows users to configure, load classes at runtime.
6. GRIB files
7. Allow bzip2 compression.
8. Netcdf Tutorial will be updated for next Unidata workshop.

International Standards:
- FeatureType Meeting in Abingdon, UK (CCLRC Rutherford Appleton Laboratory, British Atmospheric Data Center) to look at harmonizing the data models used in CDM, CSML, and OGC/ISO standards.

This document is maintained by John Caron