Hi Randy, Hi Randy, > I am a system engineer working on the GOES-R Ground Segment. This is the > next-generation GOES system that goes operation around 2015. The plan > is to use the NetCDF file format and API for the GOES-R imagery and > environmental variable product files. > > I have some general questions about the technology roadmap for NetCDF. > The current version is NetCDF4. > > (1) Is there a NetCDF5 being developed ? No, not currently. We have another format variant we are experimenting with for streaming netCDF data, but if we release it, it will still be in a netCDF 4.x version. Also the "Parallel netCDF" project at Argonne/Northwestern has developed a netCDF format variant using 64-bit sizes of things (dimensions, dimension lengths, attributes, etc.) and have named the format "netcdf-5", but we haven't integrated it into our reference distribution. > (2) If there is a NetCDF5 in works, when will the Java API be available ? The Java API typically preceeds the C/Fortran API in terms of features. The new streaming experimental format for netCDF is currently only being developed in Java. > (3) Can you provide us with your technology roadmap ? This is a small project (about 3 software engineers, on and off, each also working on other projects), and the roadmap is relatively informal. So our current plans are fluid, and periodically get revised, especially if we get proposals for further development funded. However, here's a snapshot of our current plans: === Planning for Data Access Infrastructure 2010-2014 (DRAFT) === Over the next 5 years, Unidata's data access infrastructure efforts will be challenged to * Manage a graceful transition from a simple data model (netCDF-3) to the enhanced Common Data Model of netCDF-4 * Provide better support for remote access and server-side data manipulation * Respond to the need to faithfully represent observational data as well as gridded data * Scale up to handle larger volumes of data efficiently * Provide effective support to a larger netCDF user community as users of satellite products, GIS, and other analysis and visualization software make use of growing archives of netCDF data. '''Target audience:''' Unidata will develop and maintain data access infrastructure for modelers, data providers, tool developers, and users of data to support the domestic and international geosciences research, education, and operational communities. ==== NetCDF and the Common Data Model ==== The 5 year strategy for netCDF is to improve the functionality, performance, documentation, and utility of netCDF software, while providing adequate support, backward compatibility, and interoperability with other standards. Refinement and full implementation of the Common Data Model will guide development, supporting the coordinate systems and scientific features layers in C-based APIs, reconciling differences in C and Java data models, and improving support for conventions compliance for data providers. ===== 6 months: ===== * Create netCDF-4 test file collection to support netCDF-4 tool developers, with more nested type tests (compound, vlen, and array). * 4.2 Windows port * Prepare and run netCDF workshops in October 2010. * Write guide and FAQ section for netCDF-4 chunking and compression. * NCDC project: NOMADS/NCMP TDS communication with CLASS OPeNDAP server (working with NCDC personnel) * NCDC project: aggregation of long time series from model outputs (NcML for 10 years of Climate Forecast System Reanalysis (CFSR) monthly means of single variable, GFS, NAM) * Implement and test netCDF simple record streaming as specified in standards RFC submission. * Migrate netCDF file and program examples, contributed programs, and current distribution sources to RAMADDA netCDF Group repository. * Investigate h5repack utility, nccopy enhancements for modifying chunking and compression of existing data * Enhance ncdump to optionally generate NcML output for the netCDF-4 files. * Enhance ncdump to support selection of specified groups. * Enhance ncgen4 to generate Fortran-90 output for netCDF-4 files. * Incorporate contributed pnetcdf format variant exploiting 64-bit size_t, permitting dimensions larger than 2<sup>32</sup> and no practical limits on variable size * Update [http://www.unidata.ucar.edu/netcdf/papers/nc4_conventions.html Developing Conventions for netCDF-4] document and submit what's ready to CF * NCDC project: IOSP work for ON29 BUFR, GRIB2 binary, IEEE binary * Release netCDF 4.2 C-based libraries, windows port, new netCDF-4 C++ API, bug fixes, support for tool developers * Disciplined [[9989ec31-9a6b-4943-8f79-4c16c3d9b0e0|thread safety in C-based interfaces]], assumes each thread opens file(s) separately * Upgrade Fortran support for F03 features, especially use of Fortran-to-C interfaces. * Provide examples that use new observational conventions. ===== 1 year (July 2011): ===== * Help define usable partitioning and filtering representations for DAP4. * NCDC project: support CFSR intercomparison with CMIP5 * Define canonical architectures for streaming infrastructure for CDM. * Design and document hooks in C-based netCDF to allow plug-in extensions for other formats, as supported by IOSP interfaces in netCDF-Java. * Release C-based DAP client implementation that will handle the full enhanced data model of netCDF-4. * Make all examples generate CF-compliant files. * Update and enhance attribute Users Guide conventions for netCDF-4. * Enforce that all string values must be UTF-8, checked when stored. * Hoist ncdump, ncgen, nccopy utilities to make use of common library for in-memory schema and for in-memory data block access that works for variables of any shape * Investigate extending types supported by classic format to include unsigned and 64-bit numeric types, a bounded-length string type, and an opaque type, using plug-in extensions. * Investigate whether to add remote access via http range facility to C-based software, as in netCDF-Java * Consider allowing user-jammed insertions at beginning of netCDF-4 files, like HDF5, of length 2<sup>9</sup>, 2<sup>10</sup>, 2<sup>11</sup>, ... * Register with IANA a MIME access type for netCDF, following RFC4288 and RFC4289 process. * Implement interpretation of the _Unsigned variable attribute when doing conversions of byte data. * NCDC project: IOSP work on additional formats (GHCN-monthly, GHCN-daily, USHCN, IGRA, ICOADS, IBTrACS, ...) * Utility or API to provide sizeof and offset_of for members of "dynamic" compound types, with member size determined at run-time (needed by Lynton Appel and others). ===== 2-3 years (June 2011 - June 2012): ===== * Support automatic data packing using _Scale and _Offset attributes within C library. * Finish C++ API, including use of exceptions, templates, and name spaces and full support for the CDM. * Support HDF5 references and thus reading NPOESS satellite data through netCDF interface. * With other OPeNDAP developers, investigate addition of asynchronous DAP facilities * Support enhanced streaming for optimized read access to netCDF subsets requested by coordinate system bounding boxes. * Add netCDF error info to the HDF5 error stack. * Add group support for classic format by using "/" character in names. * Create language plug-in framework for ncgen, to support output of Java, C++, Python, etc. * Modify ncgen to generate reading as well as writing code. * Support read-only access through NcML in C-based APIs, including support for subsetting, augmentation, and aggregation. * Add bit-packing support for n-bit data through HDF5 filter. * Support reading other CDM data (e.g. GRIB) through plug-in extensions for netCDF C-based API. * Write white papers on data access infrastructure, providing Unidata perspective, guidance, and leadership on specific issues. ===== 5 years (June 2014): ===== * Make the ncgen utility read NcML as well as CDL. * Improve integration with parallel netCDF for classic format files, providing merged parallel interface for netCDF-4 files also. * Make testing for other language APIs as thorough as C-based tests. * Continue to improve access to geoscience data to make it easy, reliable, flexible, and efficient. ==== Metadata conventions and libcf ==== Work must continue on developing and promoting open standards, conventions, and protocols for publishing and accessing data and metadata to enhance interoperability. Development and deployment of a library facilitating the creation and maintenance of CF-compliant data is a high priority. The library should support access to CF-compliant data that takes advantage of the CF metadata, providing data access by coordinate referenced bounding boxes, for example. Some of the advanced features in the Java interface, such as support for the scientific features layer, should be in libcf rather than the netCDF libraries. ===== 6 months: ===== * '''Done:''' Develop and submit an RFC for CF Metadata Conventions for the NASA Earth Science Standards group, referencing web site documents. * '''Done:''' Integrate Balaji's GridSpec library into libcf. ===== 1 year (December 2010): ===== * Develop and implement the Princeton API for GridSpec for use in libcf * Help AR5 modeling data centers come online with GridSpec * Get approval for CF Conventions extensions for CF observations data and coordinate reference systems. * Implement CF observations data extensions in libcf. * Implement coordinate reference systems extensionsin libcf. * Move the effort forward to agree on a CF convention for model ensembles. * Move time-date handling to libcf, from udunits. Handle climate calendars according to the CF spec. * Development of new F90 API to better support data producers. * Support CF cell methods. * Work with modeling groups to get feedback on libcf functionality. * Use package such as GDAL for implementing projection library support in libcf. * Create compliance testing utility that uses libcf. ===== 2-3 years (June 2011 - June 2012): ===== * Enhance libcf with support for namespaces and standard names. * Provide iterator-like interfaces to C-based software * Provide iterator-like interfaces to C-based software for access to variable-length data types for access to * Provide iterator-like interfaces to C-based software for access to variable-length data typesvariable-length data types * Support CF conventions for model ensembles. ===== 5 years (June 2014): ===== * Support libcf as a comprehensive reference implementation for evolving CF conventions, including support for structured grids, new observational data conventions, standard name attributes for quantities, climate modeling calendars, and interoperability with other data models. Other areas and special projects * Provide support for use of netCDF in CMIP5 model output standards for next IPCC assessment * Provide support to IOOS ocean community in use of netCDF+CF+OPeNDAP, as resources permit * Provide support for developing GOES-R ground segment product standards. * Provide support to Matlab and IDL developers for integrating netCDF-4 into Matlab and IDL * Provide feedback and support to HDF5 developers in testing releases --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: RZN-157444 Department: Support netCDF Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.