Status Report: May 2012 - Sept 2013
John Caron, Ethan Davis, Dennis Heimbigner, Marcos Hermida, Sean Arms, and Robb Kambic
Strategic Focus Areas
The THREDDS group's activities support Unidata's strategic goals in the following ways:
Enable widespread, efficient access to geoscience data
The work of the THREDDS group is comprised of two main areas: the THREDDS Data Server (TDS) and the Common Data Model (CDM) / netCDF-Java library. The TDS provides catalog and data access services for scientific data using OPeNDAP, OGC WCS and WMS, HTTP, and other remote data access protocols. The CDM provides data access through the netCDF-Java API to a variety of data formats (e.g., netCDF, HDF, GRIB). Layered above the basic data access, the CDM uses the metadata contained in datasets to provide a higher-level interface to geoscience specific features of datasets, in particular, providing geolocation and data subsetting in coordinate space. The CDM also provides the foundations for all the services made available through the TDS.
The data available from the IDD is a driving force on both the TDS and netCDF-Java development. The ability to read all the IDD data through the netCDF-Java library allows the TDS to serve that data and provide services on/for that data.
Develop and provide open-source tools for effective use of geoscience data
Unidata's Integrated Data Viewer (IDV) depends on the netCDF-java library for access to local data, and on the THREDDS Data Server (TDS) for remote access to IDD data. At the same time, the CDM depends on the IDV to validate and test CDM software. Many other tools build on the CDM / netCDF-Java library (eg ERDDAP, Panolpy, VERDI, etc) and on the TDS (LAS, ncWMS, MyOcean, etc).
Provide cyberinfrastructure leadership in data discovery, access, and use
The Common Data Model (CDM) / netCDF-Java library is one of the few general-purpose implementations of the CF (Climate and Forcast) metadata standards. Current active efforts in CF that we are involved with include draft Satellite and Radar Conventions.
The TDS has pioneered the integration of Open Geospatial Consortium (OGC) protocols into the earth science communities. Strong international collaborations have resulted in WCS and WMS services as part of the TDS.
Activities Since the Last Status Report
Release Status of CDM and TDS
- CDM 4.2.32 and TDS 4.2.10 are the current stable releases.
- CDM 4.3 is in beta release. TDS 4.3 is in alpha release. (See below for details on planned features.)
- Work continues on OPULS (OPeNDAP-Unidata Linked Servers) project to finalize DAP4 specification, implement DAP4 clients and servers, develop DAP4 conformance testing capabilities. The OPULS project recieved second year funding from NOAA -- approximately one third of originally budgeted amount.
- Working with NCAR/MMM WRF group:
- to improve CF compliance of WRF netCDF output
- on tools to allow WRF to read input from netCDF files (unnetcdf)
- on TDS file download subset service to return WRF required fields on requested space/time subset.
- on improved support of staggered grids and time dependent coordinates
- Working with NOAA/ESRL/GSD on checking the GRIB tables used with HRRR data.
- Work underway on project for "Project to Improve Software Infrastructure in Support of the NOAA National Climate Model Portal (NCMP)", funded by NOAA/NOMADS grant.
- ncSOS plug-in developed by ASA (Applied Science Associates) and USGS CIDA provides OGC SOS service for data conforming to the CF Discrete Geometries data types. Looking at distributing with the TDS version 5.0.
International Standards Development
- Track and participate in OGC MetOcean Working Group discussion.
- Track and participate in OGC WCS development.
- Track and participate in OGC CF-netCDF development.
- New Member of THREDDS Team: Lansing Madry
- Improved release engineering and development process:
- Source code repository is now on publicly visible GitHub repository.
- Build and dependency management switched to Maven (Thanks to Martin, Tom, and Marcos).
- Extending and improving unit, integration, and system testing. Using Jenkins for automated build and test (continuous integration).
- Cleaned up cyclic dependencies by refactoring several modules including grib, opendap, and VisAD-based IOSPs.
- Upgrading various external libraries, e.g., Spring 3.1, protobuf 2.4.
- Adding/improving IOSPs and CoordSysBuilders. Improving IOSP framework.
- Complete rewrite of GRIB-1 and GRIB-2 IOSPs
- Updated to latest tables from WMO: BUFR, GRIB2, Common
- Updated to latest tables from NCEP (HTML screen scrape)
- Added Coordinate System Builder for Arakawa E grid in WRF NMM core output files
- Improved IOSP registry drop-in pluggability using javax.imageio.spi.ServiceRegistry (Thanks to Tom Kunicki, USGS CIDA)
- Improved calendar handling, e.g., 360-day years for climate data.
- Cleanup projection code and add polyconic projection
- Refactor CDM/TDS use of HTTP client library and improve handling of HTTP authentication.
- New code in TDS to handle GRIB collections that can scale up to large collections of files, first application is NOMAD's CFSR dataset
- OPeNDAP: support client side certificates. Support Groups with DAP 2 protocol (TDS).
- Synchronize CDM with CF 1.6 Discrete Sampling Features (station, profiler, aircraft track data e.g.). Add new services to TDS for these data types.
- Refactor TDS catalog caching for Earth System Grid (ESG)
- Upgrade ncWMS used in TDS to latest version, 1.0RC3.
- Upgrade ncISO used in TDS to latest version, 2.0.2
- NCSS maxFileSize, time striding, allow grid subsetting by (single) vertical level
- TDS 4.3 requires Java 6 and Tomcat 6
CDM / TDS 5.0 Development
- Refactor THREDDS catalog parse/build software targeting performance and ease of use.
- Refactor TDS configuration targeting ease of use and dynamic re-initialization.
- Improved metadata extraction from datasets for use in catalogs and other metadata services.
- Improve existing and add new services for CF 1.6 Discrete Sampling Features (station, profiler, aircraft track data e.g.).
- Improve NetCDF Subset Service (NCSS)
- Experimental asynchronous / streaming / remote access services (cdmRemote)
- TDS 5.0 will require Java 7 and Tomcat 7