LEAD 1^st 7 Data Feed –  Data Descriptions

The first part of this page documents the first 7 data feeds for the Linked Environments for Atmospheric Discovery (LEAD) project as well as the first two LEAD data Products (WRF and ADAS).  The second part of this page documents the other Data Feeds being held on the Unidata LEAD Testbed system. It provides information about the raw input data types, any known decoders, transport and inventory cataloging systems for the data feed. For each data feed it also documents finally if and how the IDV and other LEAD tools currently access these data feeds.  NOTE: there may be multiple decoders and/or LEAD Tool access methods for each data type.

These data (with the exception of RADAR data) are available through the LEAD top level THREDDS catalog found at: http://lead.unidata.ucar.edu:8080/thredds/topcatalog.html (or .xml)

The Unidata portion of the above catalog is crosswalked into the LEAD resource catalog every 1/2 hour. Please see the LEAD portal at: http://portal.leadproject.org

All IDD data streams are made available through the THREDDS catalog found at: http://lead.unidata.ucar.edu:8080/thredds/catalog.html (or .xml)

 

Dataset	Raw Form	Decoded Form	Transport	Inventory List Catalog	# of Days on UPC Testbed	TDS Catalog URL
NAM Model	GRIB	NetCDF though no longer applicable for LEAD	TDS-OPeNDAP, TDS-HTTPServer, GridFTP	TDS, Resource Catalog	300	http://lead.unidata.ucar.edu:8080/thredds/lead/lead_nam_model.html (or xml) - For LEAD related holdings. http://lead.unidata.ucar.edu:8080/thredds/idd/nam_model.html (or xml) - For complete holdings.
RADAR Level II	Bzip2 (modified) Volume	Other modified Bzip2 Volume	TDS-OPeNDAP, TDS-HTTPServer	TDS	183	http://lead.unidata.ucar.edu:8080/thredds/catalog/radar2/catalog.html (or xml) - For complete holdings (NOTE: Not yet crosswalked to Resource Catalog)
RADAR Level III	NIDS	N/A	TDS-OPeNDAP, TDS-HTTPServer	TDS	200	http://lead.unidata.ucar.edu:8080/thredds/catalog/radar/catalog.html (or xml) - For complete holdings (NOTE: Not yet crosswalked to Resource Catalog)
Surface METAR Data	ASCII (WMO)	NetCDF	TDS-OPeNDAP, TDS-HTTPServer, GridFTP	TDS, Resource Catalog	600	http://lead.unidata.ucar.edu:8080/thredds/catalog/station/metar/catalog.html (or .xml)
		McIDAS MD	ADDE	N/A	N/A	N/A
		GEMPAK	N/A	N/A	N/A	N/A
Upper Air Rawinsondes	ASCII (WMO)	NetCDF	TDS-OPeNDAP, TDS-HTTPServer, GridFTP	TDS, Resource Catalog	600	http://lead.unidata.ucar.edu:8080/thredds/catalog/upperair/catalog.html (or .xml)
		McIDAS MD	ADDE	N/A	N/A	N/A
Upper Air ACARS	gzipped netCDF	N/A	N/A	N/A
GOES Satellite Imagery
WRF	NetCDF	GRIB (from WRF-POST)	TDS-OPeNDAP, TDS-HTTPServer, GridFTP	TDS, myLEAD, Siege Client	*	*
ADAS	NetCDF	N/A	TDS-OPeNDAP, TDS-HTTPServer, GridFTP	TDS, Resource Catalog

 

 

* - There are several forms of WRF data available -

EarlyLEAD work (pseudo-ensembles generated at UPC, MU and HU, steered by ADAM algorithm run at MU) found at: http://lead.unidata.ucar.edu:8080/thredds/lead/earlylead.html (or .xml)

Unidata Steered WRF runs (both NMM and ARW core) found at: http://lead.unidata.ucar.edu:8080/thredds/lead/wrf_model.html (or .xml)

LEAD Steered WRF runs from both Portal and Siege interfacing. Access to these are either through your portal account or through your Siege client.

 

There is also information on Obtaining an Upstream LDM feed and on WMO Encoding.

 

 

1) Surface METAR Data:

LDM ERE:	DDS|IDS, SA* and SP* (for surface)
Product ID:	Based on WMO encoding for TEXT products.  Unisys keeps web pages on WMO Headers for Text Data  and WMO Headers for DDPLUS.  NOAA keeps a more general page on METAR/TAF information.  Some even more general WMO references are found below in WMO Information.
Raw(IDD) form :	ASCII with WMO encodings
Decoded1:	netCDF - metar2nc and metar2nc.new1 decoder in UNIDATA NetCDF decoders
IDV Access1:	Read off local file system (i.e. not served)
ADAS Access1:	N/A?
ADaM Access1:	?
Decoded2:	McIDAS MD files -   DMSFC Data Monitor in the McIDAS XCD package
IDV Access2:	Served through ADDE
ADAS Access2:	N/A
ADaM Access2:	?
Decoded3:	GEMPAK - dcmetr decoder found in UNIDATA GEMPAK distribution
IDV Access 3:	N/A
ADAS Access 3:	Read off local file system - then it's passed through another "decoder" to put it into "LAPS" format
ADaM Access 3:	?

 

 

2) Upper Air: Rawinsondes  

LDM ERE:	DDS|IDS, U* (for upper air)
Product ID:	Based on WMO encoding for TEXT products.  Unisys keeps web pages on WMO Headers for Text Data  and WMO Headers for DDPLUS. Some even more general WMO references are found below in WMO Information.
Raw(IDD) form :	ASCII with WMO encodings
Decoded1:	NetCDF format using ua2nc decoder in UNIDATA NetCDF decoders
IDV Access1:	Read off local file system (i.e. not served)
ADAS Access1:	N/A
ADaM Access1:	?
Decoded2:	McIDAS MD files -   DMSFC Data Monitor in the McIDAS XCD package
IDV Access2:	Served through ADDE
ADAS Access2:	N/A
ADaM Access2:	?

 

3) Upper Air: ACARS

LDM ERE:      ACARS (aka PCWS)

Product ID:     Detailed information about the ACARS Product IDs is difficult to find.  Observation of the product ID feed shows a form:  FSL.CompressedNetCDF.MADIS.acars.YYYYMMDD_hhmm.gz which would appear to be the name of the file being sent.  FSL provides general ACARS Information.

Raw:                gzipped netCDF

Decoded:         Gempak surface ship format file

-         dcacars found in UNIDATA GEMPAK distribution

IDV Access:     N/A

 

 

4) RADAR: Level II aka: CRAFT  

LDM ERE:      CRAFT (aka NEXRD2)

Product ID:     The form of products on the CRAFT feed are as follows:

L2-BZIP2/RSID/YYYYMMDDHHSS/VID/PN/E

 

Where:

 

L2-BZIP2 = the literal string that precedes all Level II products

RSID = 4 Character RADAR Station ID

YYYYMMDDHHSS = Year, Month, Day, Hour, Minute and Second of Volume Scan start

VID = The Volume ID

PN = The Part Number in this volume (each part consists of 100 radials)

E = The literal 1 character string that indicates the end of the product (parts should appear in order)

 

The folks at NWS keep a RADAR Product page listing the RADAR Station IDs.

Raw:                modified BZIP2 format (1^st 4 bytes indicate remaining size)

-         Volume scans are broken into elevation sweeps which are broken into 100 radial (or fewer at end of sweep) products.           

IDV AccessRaw:   Via THREDDS Data Server (TDS)

Decoded1:       volume products also in BZIP2 format

-         nexradII decoder from CAPS

IDV Access1: Read from local disk (not served)

Decoded2:       volume products w/station info embedded (and broader OS support than nexradII decoder) also in BZIP2 fromat

-         dcnexr2 in UNIDATA GEMPACK distribution

IDV Access2:   Read from local disk (not served)

 

 

5) RADAR: Level III                       

LDM ERE:      NEXRAD (aka NNEXRAD)

Product ID:     Consists of a WMO Header (see below WMO Code Information), followed by a six-character AFOS PILS code following "/p".  Unidata maintains some information about the NEXRAD Feed.  Scroll down the page to “To specify what to do with the NEXRAD data”.   That document points to several other valuable resources.

Raw:                zlib compressed binary.

-         Content can be either polar coordinates or rasterized for display.

Decoded:         N/A

IDV Access:     Served as McIDAS AREA images by ADDE

 

 

6) Satellite: GOES – The form depends upon the feed from which the data is obtained

GOES: UNIWISC AKA: MCIDAS

LDM ERE:      MCIDAS

Product ID:     I’ve been able to find only limited documentation for this.  I did find some comments in a pqact file that appear to give some good initial insight found here.  I’ll continue to hunt for additional information.

Raw:                PNG compressed McIDAS AREA files                       

Decoded:         McIDAS Area Files

-         pnga2area found in UNIDATA LDM_McIDAS Package  

IDV Access:     Served as McIDAS AREA images by ADDE

GOES: NIMAGE

LDM ERE:      NIMAGE

Product ID:     The first part of the product id has a form of information elements separated by forward slashes (/).  This information is provided by the Unidata pqing (ingest) program. I am working on documenting that here. The second part of the product id follows the WMO form (TTAAii CCCC). The NESDIS interface control document for NOAAPORT defines the "TTAAii KNES" product schemes for the NIMAGE data feed.  It can be found at: http://www.nws.noaa.gov/noaaport/document/ then follow the link to icd_ch4 (either pdf of wpd whichever is your preferred format).  Start with section 4.1 on page 10.

Raw:                zlib compressed GINI

Decoded:         GINI

-         pngg2gini found in UNIDATA LDM_McIDAS Package

IDV Access:     Served by ADDE (note: IDV does not read GINI directly)

 

 

7) NAM Forecast Model

 

At the time of this writing, NAM Forecast Model Data is used by the LEAD project primarily for initial and boundary conditions for Regional WRF Model runs. The data is also made available for visualization to LEAD users. At this time only 80km HDS data and 40km CONDUIT data is used by the LEAD project. The CONDUIT data is used in two forms as described below.

LDM ERE:	CONDUIT (Full model output) or HDS (fewer fields)
Product ID:	Based on WMO encoding for Gridpoint data.  Unisys keeps a web page on WMO Header Information for Gridpoint Data. More general WMO references are found below in WMO Information.
Raw(IDD) form :	GRIB
IDV Access Raw:	Served by TDS OpenDAP/DODS via Common Data Model (CDM) interfacing (Note: at present IDV needs all forecast hours in a single file)
ADAS Access Raw:	Direct file access delivered via GridFTP (Note: at present ADAS needs all forecast hours in individual files with specific naming convention)
WRFSI Access:	Direct File access - delivered via GridFTP (Note: Ostensibly WRFSI can take files with multiple forecast hours, in practice it seems to prefer single forecast hour files)
	The following is no longer used by LEAD
Decoded:	NetCDF -         gribtonc found in UNIDATA NetCDF decoders

           

Other Data Sets housed on the Unidata LEAD Testbed

 

In support of the present and future needs of the LEAD project, and in support of the Unidata Community, we are also maintaining an archive of other data feeds on the Unidata LEAD Testbed system.

 

 

Dataset	Raw Form	Decoded Form	Transport	Inventory List Catalog	# of Days on UPC Testbed	TDS Catalog URL
GFS Model	GRIB	N/A	TDS-OPeNDAP, TDS-HTTPServer, WCS, NetCDF File Server, GridFTP	TDS	310	http://lead.unidata.ucar.edu:8080/thredds/idd/gfs_model.html (or xml) - For LEAD related holdings.
RUC Model	GRIB	N/A	TDS-OPeNDAP, TDS-HTTPServer, WCS, NetCDF File Server, GridFTP	TDS	220	http://lead.unidata.ucar.edu:8080/thredds/idd/ruc_model.html (or xml) - 80km Data
RUC2 Model	GRIB	N/A	TDS-OPeNDAP, TDS-HTTPServer, WCS, NetCDF File Server, GridFTP	TDS	140	http://lead.unidata.ucar.edu:8080/thredds/idd/ruc_model.html (or xml) - 20km and 40km Data

 

1) GFS Forecast Model

 

The Global Forecast System (GFS) Model Data is generated by NCEP and is described here: http://www.emc.ncep.noaa.gov/modelinfo/index.html. On the Unidata LEAD testbed we maintain archives of the following GFS datasets: Alaska 191km, CONUS 80 km, CONUS 95 km, CONUS 191 km, Global 0.5 degree, Global 2.5 degree, Hawaii 160 km, Norther Hemispheric 381km and Puerto Rico 191km.

LDM ERE:	CONDUIT (Full model output) or HRS (Regional grids)
Product ID:	Based on WMO encoding for Gridpoint data.  Unisys keeps a web page on WMO Header Information for Gridpoint Data. More general WMO references are found below in WMO Information.
Raw(IDD) form :	GRIB
IDV Access Raw:	Served by TDS OpenDAP/DODS via Common Data Model (CDM) interfacing (Note: at present IDV needs all forecast hours in a single file)

 

Obtaining Upstream IDD Feed:

For all but ACARS send e-mail to idd-connect@unidata.ucar.edu and request an upstream feed in the IDD network.   For ACARS, have your local LEAD PI send e-mail to Patty Miller at FSL: Patricia.A.Miller@noaa.gov describing your proposed use of the data (just a few sentences), and also agreeing not to redistribute the data or use it to develop commercial products or services

 

WMO Code Information

The NOAA NWS Telecommunications Operations Center provides information about International Codes & Bulletin Heading Practices.  There is detailed information about the WMO Header in particular a description and U.S. Standards on the left hand side of that page. They also maintain pages containing all the WMO Table Definitions from Manual 386 – follow the Table links on the left side of that page. The WMO maintains an experimental web site for Interactive Search of WMO Pulication No. 9 Volume C1 based on WMO header information.

 

 

Note: More detail available at: Anne's LEAD BB1 Data Page

Comments and Questions to:

Tom Baltzer at Unidata

tbaltzer@unidata.ucar.edu

303-497-8637

 

Last Updated: March 4, 2008