A joint team of developers from NCAR's Earth Observing Laboratory (EOL), NCAR's Research Applications Laboratory (RAL), and UCAR's Unidata are defining an extension to the NetCDF Climate and Forecast (CF) metadata convention. The CfRadial proposal specifies a CF-compliant NetCDF format for radar and lidar moments data in radial (i.e. polar) coordinates.
A Standard Format for Radial Data
'We wanted to create a modern radar data format that would also work for lidar data,' says Joe VanAndel of the Earth Observing Laboratory, explaining the need for a new convention. 'It's easier for researchers to share data and tools if the data is in a common format.' Current formats for files containing radar, lidar, and other radial data can be difficult to share and work with – either because the data are stored in proprietary formats or because files must be read using software that is not available on all computer platforms. Because software running on a wide variety of platforms works with Unidata's NetCDF format, basing a new format on NetCDF solves many interoperability problems.
'If you want to do exploratory work [on a dataset], then common tools like Python, Matlab, or IDL can read and visualize the data [stored in NetCDF files],' says VanAndel. 'With proprietary formats, you have to write an extension for the tool you're using just to get the data in.'
One example of this is the DORADE format, which is commonly used for radar data, but which cannot be easily read using non-specialized software tools. To help solve this problem, Mike Dixon, of NCAR's Research Applications Laboratory, has created the Radx C++ library to read, write and convert data in CfRadial, DORADE, and UF formats. Support for NEXRAD level2 and OPERA HDF5 will be added later.
Says VanAndel, 'Any instrument producing radial data could use Radx to write data in CfRadial format. Anyone writing an analysis or visualization tool could use Radx to read CfRadial data. Using Radx will make it easier to produce and consume properly formatted CfRadial data files. '
In addition to using NetCDF to ensure that data files can be read using common software packages, the CfRadial format also leverages the NetCDF Climate and Forecast metadata conventions. The CF convention defines metadata that describe what the data in each variable represent, as well as the spatial and temporal properties of the data. Adhering to these descriptive conventions allows users of disparate data sets to determine which variables are comparable, and facilitates the assimilation of data from disparate observations and models into a coherent picture.
'Getting radar data into the models is a big deal,' says Dixon. 'We're trying to work with the modeling community to agree on something common, so [researchers] can more easily use the same tools and share data.'
Extending the CF Conventions
While writing NetCDF data files that adhere to the CF convention is a good start, it doesn't solve every problem – especially for researchers collecting radial data. The CF convention was designed as a format for storing gridded data; radial data collected by instruments such as radar and lidar can be stored in a CF-compliant file, but without additional metadata fields important information about the data can be lost.
The CfRadial convention extends the CF convention by adding metadata suitable for radial data, as well as additional standard units and standard names. The new convention will make it easier to deal with data collected by mobile instruments, as well as instrument calibration data and other platform-specific information.
Dixon says that while standards can be difficult and time-consuming to create, they make it easier for investigators to store data that others can work with. VanAndel adds that part of the CfRadial effort is aimed at creating a standard that is useful to a large community, without preventing those who have additional nonstandard data from using the format. 'There will always be people who want additional capabilities [specific to their instrument],' he says. 'We are trying to keep the [CfRadial] standard simple enough to be broadly useful, without limiting people who need additional complexity.' Both VanAndel and Dixon cite the NetCDF format's ability to include additional data variables in data files without sacrificing compatibility with the CfRadial format as one of the main reasons for choosing it as the base data storage mechanism.
Involving the Community
Numerous organizations in the meteorological community have contributed resources, ideas, or otherwise expressed interest in the CfRadial project. Groups associated with NCAR/UCAR, NOAA, the Department of Energy's Atmospheric Radiation Measurement (ARM) project, the Universities of Wyoming, Oklahoma, and Wisconsin, the Center for Severe Weather Research, as well as commercial instrument makers, have provided input.
'We all have limited resources, so the more sharing we can do, the better,' says Dixon. 'If we [at UCAR] can leverage the work of other groups that may have written code for Python or Matlab or some other package, then we can devote more time to projects that we're especially well suited to tackle.'
The CfRadial development team is actively encouraging community collaboration in the project, both through the larger community involved with the CF convention and through direct contact with interested investigators and developers. As an extension to the CF convention, the CfRadial proposal is being tracked by the CF Trac system (https://cf-pcmdi.llnl.gov/trac, see proposal 59). Detailed information on the proposal and links to the development electronic mailing list are available at www.ral.ucar.edu/projects/titan/docs/radial_formats/cfradial.html. VanAndel and Dixon encourage those with an interest in radial data to submit comments and suggestions by joining the mailing list.
Dixon, VanAndel, and other members of the CfRadial development team will be presenting details of the proposed format at the 2011 meeting of the American Meteorological Society, in Seattle. The team hopes to collect additional feedback from the meteorological community and have an initial release of the convention, along with some associated software tools, available soon after the meeting.
Additional Resources
- Information on the CfRadial proposal, including the draft file format:
http://www.ral.ucar.edu/projects/titan/docs/radial_formats/cfradial.html - Information on the Climate and Forecast convention for NetCDF:
http://cf-pcmdi.llnl.gov/ - Information on the Radx radial file format conversion tools:
http://www.ral.ucar.edu/projects/titan/docs/radial_formats/radx_lib.html - Abstract of the paper to be presented at the 2011 AMS meeting:
http://ams.confex.com/ams/91Annual/webprogram/Paper178106.html
