[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20060529: Changes to NCEP Global Ensemble output (reminder)


Subject: 20060314: Changes to NCEP Global Ensemble output

CONDUIT users of ensemble data,

NCEP has notified users that the posted ensemble products will change
file naming and content organization on May 30 2006 at 12Z according to
the notice which follows. This change will affect the CONDUIT data
stream according to the identified changes to the NWS ftp server
(tgftp.nws.noaa.gov) as noted.

In addition to the changes noted regarding the file and directory names,
the result to CONDUIT is that the current fh.xxxx parameter files which
get posted in bulk (and therefore contribute to a queue backlog), will
be replaced by files which will be made available throughout the
production time of the ensemble processing on the NCEP CCS and they will
each contain multiple parameters for a single ensemble member and a
single forecast hour.

Steve Chiswell
Unidata User Support

----------------------------------------------------------------------------
*MAJOR PRODUCT CHANGES TO NCEP GLOBAL ENSEMBLE OUTPUT -- PLEASE READ
CAREFULLY*

Starting with the 12 UTC cycle on 30 May 2006 NCEP Central Operations
will implement changes to the global ensemble processing that will
result in drastic changes to its products.  These changes are necessary
to allow us to:

1. Streamline our ensemble processing
2. Provide additional ensemble members at higher resolutions on public
servers
3. Eliminate redundant global ensemble products on these servers.

These changes will impact products available on both the NCEP ftp server
(ftpprd.ncep.noaa.gov/pub/data/) and the NWS ftp server
(tgftp.nws.noaa.gov/SL.us008001/ST.opnl/).  Users of these products will
need to modify their automated retrieval and processing routines so that
they can continue to receive and use these data.

The nature of these changes are as follows:

*1.  Directory Structure - **[Affects both the NCEP and NWS ftp
servers]**
*

    The directory tree of global ensemble data will be drastically
    changed to allow for better system performance on the NCEP Central
    Computer System (CCS).  This change is being brought to the NCEP and
    NWS ftp servers as it will help with both access times and directory
    listings.

    1A. On the NCEP ftp server the directories will change as follows:

    Current -
ftpprd.ncep.noaa.gov/pub/data/nccf/com/mrf/prod/ens.{YYYYMMDD}

    New -

ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/{FILE_TYPE}

    where YYYYMMDD is the current date, CC is the cycle time (00, 06,
    12, or 18), and FILE_TYPE is a descriptor for the type of data files
    contained in the subdirectory.  Examples of FILE_TYPE that will be
    on the NCEP ftp server include: pgrba, pgrbb, ensstat, and wafs.

    Users who use the standard pressure GRIB files (ens*.pgrbf*) files
    on the NCEP ftp server will be able to access these data from one of
    the following locations:


ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrba/
    or

ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrbb/

    The exact location of the parameters you may need is given in item 3
    below.

    Users of the ensemble data on WAFS grids will be able to access
    these at:


ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/wafs/

    1B.  On the NWS ftp server (tgftp.nws.noaa.gov) the directories will
    change as follows:

    Current -

tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1/

    New -

tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1_RE.high/
            -

tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1_RE.low/

    The exact location of the parameters you may need is given in item 3
    below.


2.  *Six hourly output - [Affects both the NCEP and NWS ftp servers]*

    The global ensemble data sets will begin to be available in 6-hour
    increments out through 384 hours.  Currently 6-hour increment data
    are only available to 84 hours, then data are available in 12-hour
    increments to 384 hours.


3.  *Break Up of Pressure GRIB data sets (pgrb files) - [Affects both
the NCEP and NWS ftp servers]*

    To facilitate the removal of redundant data sets the individual
    member ensemble files are being broken up into two files per time
    step.  The first file will contain all of the parameters currently
    contained in the enspost format files (see Item 6 below) at one
    degree resolution out to 384 hours.  These are being termed the A
    files and this is an increase in their resolution for the 90 to 384
    hour time period.  The second set (the B files) will contain the
    rest of the parameters at one degree resolution to 84 hours and 2.5
    degree resolution from 90 to 384 hours.  This is the same resolution
    that these parameters are currently made available.

    On the NCEP ftp server, files will change as follows:

    Current -

ftpprd.ncep.noaa.gov/pub/data/nccf/com/mrf/prod/ens.{YYYYMMDD}/ensp1.t{CC}z.pgrbf24

    New -

ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrba/gep01.t{CC}z.pgrbaf24
    and
           -

ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrbb/gep01.t{CC}z.pgrbbf24

    On the NWS ftp server, the files will change as follows:

    Current -

tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1/fh.0024_pa.membrp01_tl.press_gr.onedeg

    New - For the "A" files:


tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1_RE.high/fh.0024_pa.membrp01_tl.press_gr.onedeg

              - For the "B" files:


tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1_RE.low/fh.0024_pa.membrp01_tl.press_gr.onedeg

    This change will allow for a subset of higher resolution (one
    degree) data to be available out to 180 hours, whereas today it is
    only available to 84 hours.  Parameter lists for the "A" and "B"
    files are located at:

    http://www.nco.ncep.noaa.gov/pmb/changes/gefs_20060530_Alist   and
    http://www.nco.ncep.noaa.gov/pmb/changes/gefs_20060530_Blist


4.  *File Names Changing due to new ensemble transform mechanism -
[Affects both the NCEP and NWS ftp servers]*

    When this change is implemented, NCEP will be creating perturbations
    in its ensemble members via a ensemble transform breeding method.
    [For specific information about this change, please see
    http://wwwt.emc.ncep.noaa.gov/gmb/yzhu/html/ENS_IMP.html]  Under
    this method, there will no longer be positive and negative pairs of
    ensemble members.  NCEP will therefore cease identifying its members
    as either positive or negative and instead use positive numbers for
    member identification.  For example, on the NCEP ftp server:

    Current -

ftpprd.ncep.noaa.gov/pub/data/nccf/com/mrf/prod/ens.{YYYYMMDD}/ensn1.t{CC}z.pgrbf24

    New -

ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrba/gep01.t{CC}z.pgrbaf24
    and/or
           -

ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrbb/gep01.t{CC}z.pgrbbf24

    The names of these files will range from gep01 to gep{NN}, where NN
    is the number of ensemble members.  A low resolution control run
    will be added for all cycles and will be named as gec00.t{CC}z.*.

    On the NWS ftp server, the names will change as follows:

    Current -

tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1/fh.0024_pa.membrn01_tl.press_gr.onedeg

    New - For the "A" files:


tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1_RE.high/fh.0024_pa.membrp01_tl.press_gr.onedeg

              - For the "B" files


tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1_RE.low/fh.0024_pa.membrp01_tl.press_gr.onedeg

    The names of these new files will range from
    fh.{FFFF}_pa.memberp01.* to fh.{FFFF}_pa.memberp{NN}.*, where NN is
    the number of ensemble members.  The control run for all cycles will
    be named as fh.{FFFF}_pa.memberc00.*.

5.  *Renaming and restructuring Ensemble mean files (enstat.*) -
**[Affects NCEP ftp server only]*

    The ensstat.* files on the NCEP ftp server will no longer be
    available.  These grids contain ensemble mean and spread information
    for a single parameter for all ensemble members and all forecast
    hours.  They will instead be replaced by files that contain multiple
    parameters for a single ensemble member and a single forecast hour.
    With this configuration the creation of these data will be made
    while the ensemble members are running on the NCEP CCS.  Currently
    these are not available until the end of the ensemble run which
    creates a large amount of data to download at once.

    The current data sets are named as follows:


ftpprd.ncep.noaa.gov/pub/data/nccf/com/mrf/prod/ens.{YYYYMMDD}/ensstat.t{CC}z.{Parameter-name}

    The new data sets will be named as follows:


ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrba/geavg.t{CC}z.pgrbaf{FFF}
    and

ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrba/gespr.t{CC}z.pgrbaf{FFF}


6.  *Removal of enspost, and ensemble Data Files - [Affects both the
NCEP and NWS ftp servers]*

    To reduce product redundancy and to allow for the additional data
    sets outlined in items number 2 and 3 above, the enspost.*
    (fh.xxxx.* files at NWS ftp) and ensemble.* data sets will no longer
    be produced.  These data sets contain redundant information that is
    already available in the pressure GRIB data sets.  They are being
    removed and users should instead use the "A" files located on the
    NCEP ftp server at:


ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrba/ge{NNN}.t{CC}z.pgrbaf{FFF}

    (where NNN is the member name, c00, p01-p10) and on the NWS ftp
    server at:


tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1_RE.high/fh.{FFFF}_pa.membr{NNN}_tl.press_gr.onedeg

    Even though these new "A" grids contain the same information as the
    enspost.* (fh.xxxx.* files at NWS ftp) and ensemble.* data sets,
    they are organized differently.  The existing files are created at
    the end of the ensemble processing and each file contains
    information for a single parameter for all ensemble members and all
    forecast hours.  The new "A" files will be made available throughout
    the production time of the ensemble processing on the NCEP CCS and
    they will each contain multiple parameters for a single ensemble
    member and a single forecast hour.

    There is one exception to this change.  The 24-hour bias corrected
    ensemble products created at 00 UTC and available only the NCEP ftp
    server, will continue to be made available in the following new
    location:


ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/ensstat/ensstat.pqpf_24h
    and ensstat.pqpf_24hbc

*7.  Corrected ensemble extensions in GRIB1 PDS - [Affects both the NCEP
and NWS ftp servers]*

    The current global ensemble data sets do not properly use the
    extended Product Definition Section (PDS) of GRIB1.  When this
    change is implemented the GRIB1 data will be correctly labeled as
    ensemble data sets.  This information is given in PDS octets 41 and
    higher.  A description of NCEP's use of these octets is given at:

    http://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html

8.  *Removal of all GRIB index files - [Affects both the NCEP and NWS
ftp servers]*

    The GRIB index files are no longer required by NCEP production and
    are being removed from the ftp servers.  These files are currently
    named as follows:

    NCEP -

ftpprd.ncep.noaa.gov/pub/data/nccf/com/mrf/prod/ens.{YYYYMMDD}/ens{NN}.t{CC}z.pgrbif24

    NWS -

tgftp.nws.noaa.gov/SL.us008001/ST.opnl/MT.ensg_CY.{CC}/RD.{YYYYMMDD}/PT.grid_DF.gr1/fh.{FFFF}i_pa.membr{NNN}_tl.press_gr.{XXX}deg

    If users still need GRIB index files, they can be created with NCEP
    software located at:

    http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/util/sorc/grbindex.fd/

9.  *Removal of the 12 UTC "e2" Member -- Addition of Low Resolution
Control Run for all Four Cycles*

    With this implementation all four cycles per day will have a low
    resolution control run.  The need for the "e2" member at 12 UTC will
    go away at that time.  The NCEP CCS resources that were used to run
    the "e2" member will be used for the low resolution control member.
    The low resolution control member data will be found at:


ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrba/gec00.t{CC}z.pgrbaf{FFF}
    and

ftpprd.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.{YYYYMMDD}/{CC}/pgrbb/gec00.t{CC}z.pgrbbf{FFF}


10.  *6-hour Accumulations for Precipitation and Maximum and Minimum
Temperature*

    In the current global ensemble system accumulation variables
    alternate between 6-hour and 12-hour amounts depending upon the
    forecast hour.  With this upgrade all accumulation amounts will be
    based on a 6-hour period.

*TEST DATA:*

At this time we are not able to provide parallel global ensemble data
sets for user to compare the old versus new.  We are however able to
provide a static data set.  These data can be viewed at:

ftpprd.ncep.noaa.gov/pub/data1/nccf/com/gens/prod/gefs.20060306/

note that these static data are located under /pub/data1 not /pub/data.

It is anticipated that we will be able to establish a real time feed of
the parallel data sets by the end of March 2006.  Once this parallel
feed of the new ensemble data sets is available, a new message will be
sent out via this mechanism.


Please direct any questions on these changes to:

address@hidden

Sincerely,

NCEP Central Operations.