Internet Data Distribution of NWS/NCEP Model Products

These data streams are provided using the Unidata LDM/IDD distribution system to create a timely and reliable delivery of products for research and education. While every attempt is made to ensure high quality data delivery, all users should note that this is not considered "Operational" data since 24 hour support is not possible, and some data may be experimental in nature. These data feeds are designed to be flexible to meet the needs of the user community, and therefore contents may change as necessary to reflect the requirements of the recipients.

More information on the CONDUIT cooperative effort between the US Weather Research Program (USWRP), NOAA's National Weather Service (NWS), NASA and Unidata to expand the distribution of model data via the IDD is available at http://www.unidata.ucar.edu/data/conduit/exp_models_how.html.

CONDUIT feed

This data feed originates at NWS/NCEP, and contains high resolution and/or research data sets available via FTP from servers at the NWS National Centers for Environmental Prediction (NCEP). Currently, approximately 42 GB of data per day (with replacement of GRIB1 with GRIB2) is transmitted on this feed. Data is placed on the LDM feed by NCEP as it is posted to the servers. Due to the nature of file posting, large amounts of data are injected at a time which should be considered when configuring local LDM performance.

IDD feed naming conventions

Model data from aggregate GRIB files is injected into the IDD as individual data products for more efficient transmission and pattern/action handling. Individual data products are identified in the LDM by the following convention:

 
 ncep_file_name !type/center/model/#grid/YYYYMMDDHHNNFfff/PARM/level! sequence 
     ^             ^     ^     ^     ^        ^        ^    ^      ^       ^
     |             |     |     |     |        |        |    |      |       |
     |             |     |     |     |    model date   |  grid     |   grid number
  data file name   |  issuing  |   grid                | parameter |   in original
  on file server   |   center  |  number               |           |   data file
                   |           |                    forecast       |
               data type   model id                  hour      grid level

Note: grib1 type products will have a "/" character before the forecast hour "F" character, while grib2 data will not. Grib2 data will provide three fields for the vertical coordinate and may contain more than one PARM identifier (multiple PARM names will be separated using a ";" character.

The ncep_file_name follows the conventions as described by the NCEP Product Services document. Click here for an example.

Any part of the product identifier can be used in the pattern/action handling regular expressions depending on how the receiving site wishes to process the data.

Since the data are injected as individual products, a status file consisting of the injection time, original file size and data file name is inserted at the end of the product sequence, with the name:
.status.ncep_file_name end_sequence The status file will contain a catalog of each individual LDM product identifier within the original data file, as well as the insertion status code returned by the LDM.

LDM pattern actions

Generic pattern/actions for data products and status messages can be used by the LDM as follows:
NCEP ftpprd.ncep.noaa.gov file names
CONDUIT	^data/nccf/com/(.*) !(.*)!
	FILE	data/conduit/\1
CONDUIT	^.status\.(.*) [0-9][0-9][0-9][0-9][0-9][0-9]
	FILE	-close	data/conduit/status/\1

TIGGE specific ensemble products
CONDUIT ^(data2/TIGGE)/(.*) !(.*)/(.*)/(.*)/(#.*)/(.*)F(...)/(.*)! (.*)
        FILE    data/conduit/\1/\7/\2
CONDUIT ^.status\.(data2/TIGGE)/(.*)\.t([0-2][0-9])z\.(.*) [0-9][0-9][0-9][0-9][0-9][0-9]
        FILE    -close  data/conduit/status/\1/%Y%m%d\300/\2.t\3z.\4

LDM Configuration

The maximum product queue usage for the CONDUIT feed is approximately 4-5 GB, or generally the greatest amount of data expected to be received in 1 hour.

Data stream contents

Representative product inventories for models.

CONDUIT feed monitoring statistics and archive retrieval

Current IDD Routing Topology

IDD rtstats from participating hosts

ftpprd.ncep.noaa.gov Products received in past 72 hours

Reference Information