[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Motherlode clone.



Chris and Mike,

sorry for the delay in getting back to you about the pqacts, etc.

there are three thredd pqacts that i'll attach: pqact.thredds
pqact.threddsconduit  pqact.threddsradar that need to be place in ~ldm/etc
directory. also the etc/ldmd.conf file needs to have these three line
added to reference the pqacts.

exec    "pqact -f ANY-CONDUIT-NNEXRAD-CRAFT /opt/ldm/etc/pqact.thredds"
exec    "pqact -f CONDUIT|SPARE /opt/ldm/etc/pqact.threddsconduit"
exec    "pqact -f NNEXRAD|CRAFT /opt/ldm/etc/pqact.threddsradar"


i believe the only extra s/w needed is the netCDF decoders package, it's
available at: http://www.unidata.ucar.edu/software/decoders/

the binaries from the build go into the ~ldm/decoders directory and the
etc files from the build go into the ~ldm/etc directory.

if you want to bypass this step, comment out the pqact
entries that have the PIPE actions then you can come back later to install
them.


there is a new perl script used to scour the directory structure called
manageFilesDirs.pl and it goes into the ~ldm/util directory. i'll attach
the script too.  also there is a manageFilesDirs.conf file to configure it
that goes into the ~ldm/etc direction, i'll attach too.  the conf file
should be set correctly, so no mods will be needed unless you change the
directory structure in the pqacts. i suggest that you don't change them
because all the thredds configure file exepect that dir structure.

it is run once a day from cron, ie

#
# files/dirs managing script
30 21 * * *  util/manageFilesDirs.pl -f
/local/ldm/etc/manageFilesDirs.conf > /data/ldm/logs/manageFilesDirs.log
#


after this is setup, then we can start on the thredds configurations.

robb...


On Tue, 23 May 2006, Chris Herbster wrote:

> Rob,
>
> We seem to be ready on this end for a set of pqact entries to populate
> the catalog.  Please work with Mike to see if he needs any other parts
> besides these.  I think he said something about needing some XML pieces too.
>
> Thanks!
>
> CH
>
> Robb Kambic wrote:
>
> >Sounds good to me, let me know when help is needed?
> >
> >robb...
> >
> >
> >
> >
> >On Mon, 24 Apr 2006, Chris Herbster wrote:
> >
> >
> >
> >>Hi Robb,
> >>
> >>As always, another good time in Boulder!  (-;
> >>
> >>Off hand, I'd say, "Let's do everything that you do."  However, I'll
> >>defer on this to Mike Masscotte.  He has done a great job of getting
> >>things cleaned up from when I tried to run things and he might have some
> >>suggestions (or just say "Let's do it!")
> >>
> >>We have the LDM up and running.  We'll need to do the next piece to get
> >>ready....  I'll let Mike set the pace on this end.
> >>
> >>CH
> >>
> >>Robb Kambic wrote:
> >>
> >>
> >>
> >>>Hey Chris,
> >>>
> >>>It was really good to sit down and talk with you at usercomm diner. I
> >>>haven't had that much fun in awhile. Anyway, if you are still interested
> >>>in doing a motherlode clone we can start on it. Could you list the data
> >>>products that you are interested in receiving? or are you interested in
> >>>receiving all products just like motherlode?
> >>>
> >>>
> >>>Once the initial setup is done, then upgrades should be easy. The 2 main
> >>>external programs are the LDM and Jakarta-Tomcat. If you could get your
> >>>system admin to install these with users ldm and tomcat then stage would be
> >>>set. The tomcat version we are using is jakarta-tomcat-5.0.28, i know
> >>>there are newer ones released but this is the version that we have tested
> >>>the s/w.  At this point do you want to install the s/w, there are on-line
> >>>documentation or do you want me to install it and do the initial
> >>>configuration?
> >>>
> >>>The documentation is located at:
> >>>
> >>>http://motherlode.ucar.edu:8080/thredds/docs/
> >>>
> >>>
> >>>Take care,
> >>>
> >>>robb...
> >>>
> >>>
> >>>
> >>>===============================================================================
> >>>Robb Kambic                                   Unidata Program Center
> >>>Software Engineer III                         Univ. Corp for Atmospheric 
> >>>Research
> >>>address@hidden                WWW: http://www.unidata.ucar.edu/
> >>>===============================================================================
> >>>
> >>>
> >>>
> >>>
> >>>
> >>--
> >>
> >> Dr. Christopher G. Herbster
> >> Associate Professor
> >> Director of Science and Technology
> >> for the ERAU Weather Center
> >> Applied Aviation Sciences
> >> Embry-Riddle Aeronautical Univ.
> >> 600 S. Clyde Morris Blvd.
> >> Daytona Beach, FL 32114-3900
> >>
> >> 386.226.6444 Office
> >> 386.226.6446 Weather Center
> >> http://wx.erau.edu/
> >>
> >>
> >>
> >>
> >
> >===============================================================================
> >Robb Kambic                             Unidata Program Center
> >Software Engineer III                           Univ. Corp for Atmospheric 
> >Research
> >address@hidden                  WWW: http://www.unidata.ucar.edu/
> >===============================================================================
> >
> >
> >
>
> --
>
>  Dr. Christopher G. Herbster
>  Associate Professor
>  Director of Science and Technology
>  for the ERAU Weather Center
>  Applied Aviation Sciences
>  Embry-Riddle Aeronautical Univ.
>  600 S. Clyde Morris Blvd.
>  Daytona Beach, FL 32114-3900
>
>  386.226.6444 Office
>  386.226.6446 Weather Center
>  http://wx.erau.edu/
>
>
>

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================
#
# This pqact defines the directory structure expected by the THREDDS Data Server
# default configuration.  It includes the pqact entries using decoders and FILE
# actions to create the structure. If this pqact is implemented then the 
# THREDDS Data Server should work without doing any data structure 
# configurations. Default configuration starting at the ~ldm/data directory:
#
# pub
# pub/native
#
# data/pub/native/surface/metar/{day}/{station}
# data/pub/native/surface/synoptic/{day}/{station} commented out
# data/pub/native/surface/buoy/{day}/{station} commented out
# data/pub/native/surface/zone/{day}/{station} commented out
# data/pub/native/nwstg/text/{day}
# data/pub/native/upperair/{day}/{station} commented out
#
# pub/native/satellite/{band}/{product}_{resolution}/{day}
#
# pub/native/profiler/wind/06min/{day}
# pub/native/profiler/wind/01hr/{day}
# pub/native/profiler/RASS/06min/{day}
# pub/native/profiler/RASS/01hr/{day}
# pub/native/profiler/bufr/{day}  commented out
#
# pub/native/radar/10km_mosaic/{day}
# pub/native/radar/composite/{product}/{resolution}/{day}
# pub/native/radar/level2/{station}/{day}
# pub/native/radar/level3/{product}/{station}/{day}
#
# pub/native/grid/NCEP/DGEX/Alaska_12km
# pub/native/grid/NCEP/DGEX/CONUS_12km //Downscaled GFS with Eta Extension(DGEX)
# pub/native/grid/NCEP/GFS/Alaska_191km
# pub/native/grid/NCEP/GFS/CONUS_80km
# pub/native/grid/NCEP/GFS/CONUS_95km
# pub/native/grid/NCEP/GFS/CONUS_191km
# pub/native/grid/NCEP/GFS/Ensemble_1p25deg
# pub/native/grid/NCEP/GFS/Extended_5p0deg
# pub/native/grid/NCEP/GFS/Global_0p5deg
# pub/native/grid/NCEP/GFS/Global_onedeg
# pub/native/grid/NCEP/GFS/Global_2p5deg
# pub/native/grid/NCEP/GFS/Global_1p0deg_Ensemble
# pub/native/grid/NCEP/GFS/Global_2p5deg_Ensemble
# pub/native/grid/NCEP/GFS/Hawaii_160km
# pub/native/grid/NCEP/GFS/N_Hemisphere_381km
# pub/native/grid/NCEP/GFS/Puerto_Rico_191km
# pub/native/grid/NCEP/GFS/Spectral_1p25deg
# pub/native/grid/NCEP/NAM/Alaska_11km
# pub/native/grid/NCEP/NAM/Alaska_22km
# pub/native/grid/NCEP/NAM/Alaska_45km/conduit
# pub/native/grid/NCEP/NAM/Alaska_45km/noaaport
# pub/native/grid/NCEP/NAM/Alaska_95km
# pub/native/grid/NCEP/NAM/CONUS_12km
# pub/native/grid/NCEP/NAM/CONUS_20km/noaaport
# pub/native/grid/NCEP/NAM/CONUS_20km/surface
# pub/native/grid/NCEP/NAM/CONUS_20km/selectsurface
# pub/native/grid/NCEP/NAM/CONUS_40km/noaaport
# pub/native/grid/NCEP/NAM/CONUS_40km/conduit
# pub/native/grid/NCEP/NAM/CONUS_80km
# pub/native/grid/NCEP/NAM/DGEX/Alaska_12km
# pub/native/grid/NCEP/NAM/DGEX/CONUS_12km
# pub/native/grid/NCEP/NAM/Polar_90km
# pub/native/grid/NCEP/NDFD/CONUS_5km
# pub/native/grid/NCEP/OCEAN/Global_2x2deg
# pub/native/grid/NCEP/RUC/CONUS_40km
# pub/native/grid/NCEP/RUC/CONUS_80km
# pub/native/grid/NCEP/RUC2/CONUS_20km/hybrid
# pub/native/grid/NCEP/RUC2/CONUS_20km/pressure
# pub/native/grid/NCEP/RUC2/CONUS_20km/surface
# pub/native/grid/NCEP/SPECTRAL/Global_5x2p5deg
# pub/native/grid/ECMWF/model?
# pub/native/grid/UKMET/thin
# pub/native/grid/CMC/GEM                // Canadian grid
# pub/native/grid/FNMOC/NOGAPS             // FNMOC grid
# pub/native/grid/FNMOC/COAMPS             // FNMOC grid
# pub/native/grid/UNIDATA/WSETA
# pub/native/grid/{center}/WRF
#
# grids off of nwstg binary channel, entered above in correct model location
# pub/native/grid/NCEP/NAM/CONUS_12km
# pub/native/grid/NCEP/NAM/Alaska_11km
# pub/native/grid/NCEP/DGEX/CONUS_12km //Downscaled GFS with Eta Extension(DGEX)
# pub/native/grid/NCEP/DGEX/Alaska_12km
# pub/native/grid/NCEP/NDFD/CONUS_5km
# pub/native/grid/NCEP/GFS/CONUS_191km
#
# Decoded data
#
# pub/decoded/netcdf/surface/metar
# pub/decoded/netcdf/surface/synoptic
# pub/decoded/netcdf/surface/buoy
# pub/decoded/netcdf/upperair
# pub/decoded/netcdf/grid/{center}
# pub/decoded/netcdf/grid/NCEP/GFS/CONUS_80km
# pub/decoded/netcdf/grid/NCEP/GFS/Global_5x2p5deg
# pub/decoded/netcdf/grid/NCEP/GFS/Global_5x5deg
# pub/decoded/netcdf/grid/NCEP/NAM/CONUS_80km
# pub/decoded/netcdf/grid/NCEP/NGM/CONUS_80km
# pub/decoded/netcdf/grid/NCEP/OCEAN/Global_5x2p5deg
# pub/decoded/netcdf/grid/NCEP/RUC/CONUS_80km
# pub/decoded/netcdf/grid/NCEP/RUC2/CONUS_40km
# pub/decoded/netcdf/grid/NCEP/SST/Global_2x2deg
# pub/decoded/netcdf/grid/NCEP/SST/Global_5x2p5deg
#
#
# Restricted data
#
# ucar_only/native/ACARS/{day} 
#
#-------------------------------------------------------------------------------
#
# Surface test data
#
# All aviation reports including metar tests, broken/written to stn files
DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
        PIPE    decoders/metarWriter
        data/pub/native/surface/metar
        (\2:yyyy)(\2:mm)
#
# Synoptic reports, broken/written to stn files
#DDS|IDS        ^SM.... .... ([0-3][0-9])
#       PIPE    decoders/synopticWriter
#       data/pub/native/surface/synoptic
#       (\1:yyyy)(\1:mm)
#
# buoy reports, broken/written to stn or WMO ID numbers files
#DDS|IDS        ^S[NS].... .... ([0-3][0-9])
#       PIPE    decoders/buoyWriter
#       data/pub/native/surface/buoy
#       (\1:yyyy)(\1:mm)
#
# zone reports, broken/written to zone files
#DDS|IDS        ^FOUS5. .... ([0-3][0-9])
#       PIPE    decoders/surface/zoneWriter
#       data/pub/native/zone
#       (\1:yyyy)(\1:mm)
#
# all of DDS/PPS/IDS | NOAAPORT textual data
DDPLUS|IDS      ^[^\-]
        FILE
        data/pub/native/nwstg/text/%Y%m%d/SURFACE_DDPLUS_%Y%m%d_%H00.txt
#
# upper air reports, broken/written to stn files
#DDS|IDS        ^U[ABDEFGHIJKLMPQRSTXZ].... .... ([0-3][0-9])
#       PIPE    decoders/upperairWriter
#       data/pub/native/upperair
#       (\1:yyyy)(\1:mm)
#
#
# NOAAPORT GINI Images
#
# PNG compressed NOAAPORT GINI Images
#NIMAGE ^sat/ch[0-9]/.*/(.*)/([12][0-9][0-9][0-9][01][0-9][0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
#       FILE
#       data/pub/native/satellite/\1/\5_\6/\2/\5_\6_\1_\2_\3\4.gini
#
#       PIPE    -close
#       util/pngg2gini -vl logs/ldm-mcidas.log
#
# Zlib compressed NOAAPORT GOES-East/West GINI Images -- FILE
NIMAGE  ^satz/ch[0-9]/.*/(.*)/([12][0-9][0-9][0-9][01][0-9][0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        PIPE    -close
        util/ldmfile.sh 
data/pub/native/satellite/\1/\5_\6/\2/\5_\6_\1_\2_\3\4.gini
#
# FSL hourly summary and 6-minute wind profiler data %Y%m%d_%H00
FSL2    ^FSL\.NetCDF\.NOAAnet\.windprofiler\.(01hr|06min)\.(....)(...)(..)
        FILE
        data/pub/native/profiler/wind/\1/\2%m%d/PROFILER_wind_\1_\2%m%d_\400.nc
#
# FSL hourly summary and 6-minute RASS profiler data %Y%m%d_%H00
# FSL.NetCDF.NOAAnet.RASS.06min.20052731424
FSL2    ^FSL\.NetCDF\.NOAAnet\.RASS\.(01hr|06min)\.(....)(...)(..)
        FILE
        data/pub/native/profiler/RASS/\1/\2%m%d/PROFILER_RASS_\1_\2%m%d_\400.nc
#
# profiler data in BUFR format
HRS     ^(IUPT0[1-4]|IUAK01) .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/profiler/bufr/profiler/PROFILER_(\2:yyyy)(\2:mm)\2_0000.bufr
#
#
# profiler data in BUFR format
HRS     ^IUPT41 KBOU ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/profiler/bufr/profiler1/PROFILER_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# profiler data in BUFR format
HRS     ^IUPT42 KBOU ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/profiler/bufr/profiler2/PROFILER_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# profiler data in BUFR format
HRS     ^IUPT43 KBOU ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/profiler/bufr/profiler3/PROFILER_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# profiler data in BUFR format
HRS     ^ISAT.. KBOU ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/profiler/bufr/profiler4/PROFILER_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# profiler data in BUFR format
HRS     ^IUPC0[12] .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/profiler/bufr/profiler5/PROFILER_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# RADAR grid products
#
HRS     ^HAXA00 K... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/radar/10km_mosaic/(\1:yyyy)(\1:mm)\1/RADAR_10km_mosaic_(\1:yyyy)(\1:mm)\1_\2\3.grib1
#
# DPA (SDUS8.)
HRS     ^SDUS8. (....) ([0-3][0-9])([0-2][0-9])([0-9][0-9]).*/pDPA
        FILE
        
data/pub/native/radar/level3/DPA/\1/Level3_\1_DPA_(\2:yyyy)(\2:mm)\2_0000.dpa
#       
data/pub/native/radar/level3/DPA/\1/(\2:yyyy)(\2:mm)\2/Level3_\1_DPA_(\2:yyyy)(\2:mm)\2_0000.dpa
#
# RCM (SDUS4.)
#HRS    ^SDUS4. (....) ([0-3][0-9])([0-2][0-9])([0-9][0-9]).*/pRCM
#       FILE
#       
data/pub/native/radar/level3/RCM/\1/Level3_\1_RCM_(\2:yyyy)(\2:mm)\2_0000.rcm
#       
data/pub/native/radar/level3/RCM/\1/(\2:yyyy)(\2:mm)\2/Level3_\1_RCM_(\2:yyyy)(\2:mm)\2_0000.rcm
#
# 1 km Regional N0R Composite - Creates PNG-compressed AREA for FNEXRAD feed
#FNEXRAD        ^radar_mosaic_regional_(...)
#       PIPE    -close
#       util/mcgribdec \1 
/data/ldm/pub/native/radar/national/radar_regional.grib1 6100
#
# 6 km National N0R Composite - Creates PNG-compressed AREA for FNEXRAD feed
#FNEXRAD        ^radar_mosaic_national
#       PIPE    -close
#       util/mcgribdec NAT 
/data/ldm/pub/native/radar/national/radar_national.grib1 6200
#
# 10 km National RCM Composite - Creates PNG-compressed AREA for FNEXRAD feed
#HRS    ^HAXA00 K...
#       PIPE    -close
#       util/mcgribdec RCM 
/data/ldm/pub/native/radar/national/radar_national_rcm.grib1
#
#
# Model files in grib1 format
#
# GFS/AVN grid output
# Grid #201 N. Hemisphere 381km: ^Y.A... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Grid #202 CONUS         191km: ^Y.I... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Grid #203 Alaska        191km: ^Y.J... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Grid #211 CONUS          80km: ^Y.Q... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Grid #213 CONUS          95km: ^Y.H... KWBC.*/m(AVN|GFS|SSIAVN|SSIGFS)
# Select any/all grids desired from [QAIJH]
#
HRS     ^[YZ].(A)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(AVN|GFS|SSIAVN|SSIGFS)
        FILE
        
data/pub/native/grid/NCEP/GFS/N_Hemisphere_381km/GFS_N_Hemisphere_381km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(I)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(AVN|GFS|SSIAVN|SSIGFS)
        FILE
        
data/pub/native/grid/NCEP/GFS/CONUS_191km/GFS_CONUS_191km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(J)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(AVN|GFS|SSIAVN|SSIGFS)
        FILE
        
data/pub/native/grid/NCEP/GFS/Alaska_191km/GFS_Alaska_191km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(Q)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(AVN|GFS|SSIAVN|SSIGFS)
        FILE
        
data/pub/native/grid/NCEP/GFS/CONUS_80km/GFS_CONUS_80km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(H)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(AVN|GFS|SSIAVN|SSIGFS)
        FILE
        
data/pub/native/grid/NCEP/GFS/CONUS_95km/GFS_CONUS_95km_(\2:yyyy)(\2:mm)\2_\300.grib1
#
# GFS grid on thinned grids, global 1.25 regular grid
HRS     ^H.[I-P]... KWB. (..)(..).*/m(AVN|SSIAVN|GFS|SSIGFS)
        FILE
        
data/pub/native/grid/NCEP/GFS/Global_1p25deg/GFS_Global_1p25deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# Spectral Global grid - made up of grids I-P that are thinned grids
HRS     ^H.[I-P]... KWB[^K] ([0-3][0-9])([0-2][0-9])
        FILE
        
data/pub/native/grid/NCEP/GFS/Spectral_1p25deg/GFS_Spectral_1p25deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
#Ensemble members
HRS     ^H.[I-P]... KWBK ([0-3][0-9])([0-2][0-9])
        FILE
        
data/pub/native/grid/NCEP/GFS/Ensemble_1p25deg/GFS_Ensemble_1p25deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# Grid #204 Hawaii        160km: ^Y.K... KWBH
# Grid #205 Puerto Rico   191km: ^Y.L... KWBH
#
HRS     ^[YZ].(K)... KWBH ([0-3][0-9])([0-2][0-9])
        FILE
        
data/pub/native/grid/NCEP/GFS/Hawaii_160km/GFS_Hawaii_160km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(L)... KWBH ([0-3][0-9])([0-2][0-9])
        FILE
        
data/pub/native/grid/NCEP/GFS/Puerto_Rico_191km/GFS_Puerto_Rico_191km_(\2:yyyy)(\2:mm)\2_\300.grib1
#
#  GFS grid - Extended range grids E and F
#
HRS     ^H.[EF][A-Z][0-9][0-9] KWB. ([0-3][0-9])([0-2][0-9])..
        FILE
        
data/pub/native/grid/NCEP/GFS/Extended_Global_5p0deg/GFS_Extended_Global_5p0deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# ETA/NAM grids
# Grid #207 95km Alaska:   ^[YZ].N.*/m(ETA|NAM)
# Grid #211 80km CONUS:    ^[YZ].Q.*/m(ETA|NAM)
# Grid #212 40km CONUS:    ^[YZ].R.*/m(ETA|NAM)
# Grid #215 20km CONUS:    ^[YZ].U.*/m(ETA|NAM)
# Grid #216 45km Alaska:   ^[YZ].V.*/m(ETA|NAM)
# Grid #217 22km Alaska:   ^[YZ].Y.*/m(ETA|NAM)
# Select any/all grids desired from [QRUT]
HRS     ^[YZ].(N)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(ETA|NAM)
        FILE
        
data/pub/native/grid/NCEP/NAM/Alaska_95km/NAM_Alaska_95km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(Q)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(ETA|NAM)
        FILE
        
data/pub/native/grid/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(R)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(ETA|NAM)
        FILE
        
data/pub/native/grid/NCEP/NAM/CONUS_40km/noaaport/NAM_CONUS_40km_noaaport_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(U)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(ETA|NAM)
        FILE
        
data/pub/native/grid/NCEP/NAM/CONUS_20km/noaaport/NAM_CONUS_20km_noaaport_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(V)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(ETA|NAM)
        FILE
        
data/pub/native/grid/NCEP/NAM/Alaska_45km/noaaport/NAM_Alaska_45km_noaaport_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(Y)... KWB. ([0-3][0-9])([0-2][0-9]).*/m(ETA|NAM)
        FILE
        
data/pub/native/grid/NCEP/NAM/Alaska_22km/NAM_Alaska_22km_(\2:yyyy)(\2:mm)\2_\300.grib1
#
# RUC grid output
# Grid #211 CONUS   80km: ^[YZ].Q.*/mRUC
# Grid #236 CONUS   40km: ^[YZ].W.*/mRUC
#
HRS     ^[YZ].(Q)... KWB. ([0-3][0-9])([0-2][0-9]).*/mRUC
        FILE
        
data/pub/native/grid/NCEP/RUC/CONUS_80km/RUC_CONUS_80km_(\2:yyyy)(\2:mm)\2_\300.grib1
HRS     ^[YZ].(W)... KWB. ([0-3][0-9])([0-2][0-9]).*/mRUC
        FILE
        
data/pub/native/grid/NCEP/RUC/CONUS_40km/RUC_CONUS_40km_(\2:yyyy)(\2:mm)\2_\300.grib1
#
# RUC2 grid on Lambert conformal CONUS grid
HRS     ^Y.W... KWB. (..)(..).*/mRUC2
        FILE
        
data/pub/native/grid/NCEP/RUC2/CONUS_40km/RUC2_CONUS_40km_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# Spectral Global grid - made up of grids A,B,C,D Forecasts (wind, waves, temp)
HRS     ^H[A-Z][ABCD][A-Z][0-9][0-9] KWB. ([0-3][0-9])([0-2][0-9])..
        FILE
        
data/pub/native/grid/NCEP/SPECTRAL/Global_5x2p5deg/SPECTRAL_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# ocean grid
#
HRS     ^H[HKLTUV][T-W][ACEIK]93 KWB. ([0-3][0-9])([0-2][0-9])
        FILE
        
data/pub/native/grid/NCEP/OCEAN/Global_2x2deg/OCEAN_Global_2x2deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# NOW and PWAV grids on global 5.0 x 2.5 degree grids
HRS     ^H.[A-D]... .... (..)(..).*(/mNOW|/mPWAV)
        FILE
        
data/pub/native/grid/NCEP/OCEAN/Global_5x2p5deg/OCEAN_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# SST grid on global 5.0 x 2.5 degree grids
HRS     ^H.[A-D]... .... (..)(..).*/mSST
        FILE
        
data/pub/native/grid/NCEP/SST/Global_5x2p5deg/SST_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# SST grid on global 2.0 x 2.0 degree grid
HRS     ^H.[T-W]... .... (..)(..).*/mSST
        FILE
        
data/pub/native/grid/NCEP/SST/Global_2x2deg/SST_Global_2x2deg_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# ECMWF grid on global 2.5 x 2.5 degree grid
HRS     ^H.[A-L]... ECMF (..)(..)
        FILE
        
data/pub/native/grid/ECMWF/Global_2p5/ECMWF_Global_2p5_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# ECMWF GRIB products, decode into yymmdd_ecmf#.gem (e1=global,e2=+/-35lat)
#
#HRS    ^H[RHPT][A-L][A-Z][0-9][0-9] ECMF ([0-3][0-9])([0-2][0-9])
#       FILE
#       data/pub/native/grid/ECMWF/model_xx/ECMWF1_(\1:yyyy)(\1:mm)\1_\200.grib1
#
#HRS    ^H[UV][A-L][A-Z][0-9][0-9] ECMF ([0-3][0-9])([0-2][0-9])
#       FILE
#       data/pub/native/grid/ECMWF/model_xx/ECMWF2_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# Global UKMet thinned grids
HRS     ^H.[I-P]... EGRR ([0-3][0-9])([0-2][0-9])
        FILE
        data/pub/native/grid/UKMET/thin/UKMET_(\1:yyyy)(\1:mm)\1_\200.grib1
#
# other soundings and surface products in BUFR format
#
# NCEP NAM MODEL BUFR SOUNDING
# JUSA41 JUSA42 JUSB43 JUSB44 JUSB45 JUSB46 JUSX47 ? JUSX48 JUSX49 KWNO
HRS     ^JUS.41 KWNO ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/soundings/NAM41/SoundingNAM_(\1:yyyy)(\1:mm)\1_\200.bufr
HRS     ^JUS.42 KWNO ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/soundings/NAM42/SoundingNAM_(\1:yyyy)(\1:mm)\1_\200.bufr
HRS     ^JUS.43 KWNO ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/soundings/NAM43/SoundingNAM_(\1:yyyy)(\1:mm)\1_\200.bufr
HRS     ^JUS.44 KWNO ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/soundings/NAM44/SoundingNAM_(\1:yyyy)(\1:mm)\1_\200.bufr
HRS     ^JUS.45 KWNO ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/soundings/NAM45/SoundingNAM_(\1:yyyy)(\1:mm)\1_\200.bufr
HRS     ^JUS.46 KWNO ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/soundings/NAM46/SoundingNAM_(\1:yyyy)(\1:mm)\1_\200.bufr
HRS     ^JUS.47 KWNO ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/soundings/NAM47/SoundingNAM_(\1:yyyy)(\1:mm)\1_\200.bufr
HRS     ^JUS.48 KWNO ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/soundings/NAM48/SoundingNAM_(\1:yyyy)(\1:mm)\1_\200.bufr
HRS     ^JUS.49 KWNO ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/soundings/NAM49/SoundingNAM_(\1:yyyy)(\1:mm)\1_\200.bufr
#
# NCEP GFS MODEL BUFR SOUNDING
# JUSA41 JUSA42 JUSB43 JUSB44 JUSB45 JUSB46 JUSX47 JUSX48 JUSX49 KWBC
HRS     ^JUS.4. KWBC ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/GFS/SoundingVerticalGFS_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# Surface data (satellite)
#
HRS     ^ISXT.. KBOU ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/surface/satellite/SurfaceSatellite_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# Surface data (satellite)
#
HRS     ^(ISXX0.|FPAK79|SXUS..|SFUS41|SACN..|PEIK98) .... 
([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/surface/satellite1/SurfaceSatellite_(\2:yyyy)(\2:mm)\2_0000.bufr
#
# Surface data (satellite)
#
HRS     ^ISZ.59 .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/surface/satellite2/SurfaceSatellite_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Surface data (satellite)
#
HRS     ^ISXA.. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/surface/satellite3/SurfaceSatellite_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# Surface data (satellite)
#
HRS     ^(ISXX0.|FPAK79|SXUS..|SFUS41|SACN..|PEIK98) .... 
([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/surface/satellite1/SurfaceSatellite_(\2:yyyy)(\2:mm)\2_0000.bufr
#
# Single level upper-air data (other than satellite)
#
HRS     ^IUA... (CWAO|EGRR) ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/singlelevel/aircraft/SoundingSingleLevelAircraft_(\2:yyyy)(\2:mm)\2_0000.bufr
#
# Vertical soundings (other than satellite)
HRS     ^IUS[TX].[68] KWBC ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/vertical/radiosonde8/SoundingVerticalRadiosonde_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# Vertical soundings (other than satellite)
HRS     ^IUS[TX].2 KWBC ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/vertical/radiosonde2/SoundingVerticalRadiosonde_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# Vertical soundings (other than satellite)
HRS     ^IUS[TX].[13] KWBC ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/vertical/radiosonde3/SoundingVerticalRadiosonde_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# Vertical soundings (other than satellite)
HRS     ^IUS[TX].4 KWBC ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/vertical/radiosonde4/SoundingVerticalRadiosonde_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Single level upper-air data (satellite)
HRS     ^J.CX.1 KNES ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/singlelevel/satellite/SoundingSingleLevelSatellite_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# Single level upper-air data (satellite)
HRS     ^(IUCN[45678]|IUCS[5678]|JHCX1|JRCX7). .... 
([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/singlelevel/satellite1/SoundingSingleLevelSatellite_(\2:yyyy)(\2:mm)\2_0000.bufr
#
# Surface data - land
#
HRS     ^JSAT.. KKCI ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/surface/land/SurfaceLand_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Vertical soundings (satellite)
HRS     ^JUTX.. KNES ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/vertical/satellite/SoundingVerticalSatellite_(\1:yyyy)(\1:mm)\1_0000.bufr
#
#
# Vertical soundings (satellite)
HRS     ^IUTX.. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/vertical/satellite1/SoundingVerticalSatellite_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Vertical soundings (other than satellite)
HRS     ^IXUS68 PACR ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/soundings/vertical/wind_profiler/SoundingVerticalWindProfiler_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Synoptic data feature 7
HRS     ^JUVE.. KKCI ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/synoptic/feature7/SynopticFeature7_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Synoptic data feature 6
HRS     ^JUCE.. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/synoptic/feature6/SynopticFeature6_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Synoptic data feature 5
HRS     ^JUFE.. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/synoptic/feature5/SynopticFeature5_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Synoptic data feature 4
HRS     ^(YUZI96 KWBC|JUBE99 ....) ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/synoptic/feature4/SynopticFeature4_(\2:yyyy)(\2:mm)\2_0000.bufr
#
# Synoptic data feature 3
HRS     ^JUVE.. EGRR ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/synoptic/feature3/SynopticFeature3_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Synoptic data feature 2
HRS     ^JUTE.. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/synoptic/feature2/SynopticFeature2_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Synoptic data feature 1
HRS     ^JUWE.. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        
data/pub/native/synoptic/feature1/SynopticFeature1_(\1:yyyy)(\1:mm)\1_0000.bufr
#
HRS     ^JUVE.. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/other/JUV/JUV_(\1:yyyy)(\1:mm)\1_0000.bufr
#
HRS     ^IUPC0. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/other/IUPC/IUPC_(\1:yyyy)(\1:mm)\1_0000.bufr
#
HRS     ^JSMT.. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/other/JSMT/JSMT_(\1:yyyy)(\1:mm)\1_0000.bufr
#
HRS     ^JU[JOMN]E00 .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/other/JUE/JUE_(\1:yyyy)(\1:mm)\1_0000.bufr
#
HRS     ^JSML1. .... ([0-3][0-9])([0-2][0-9])([0-9][0-9])
        FILE
        data/pub/native/other/JSML1/JSML1_(\1:yyyy)(\1:mm)\1_0000.bufr
#
# Other NMC assorted binary products
HRS     ^_ELSE_$
        FILE
        data/pub/native/other/hrs/%Y%m%d_%H_HRS.wmo
#
# CMC GEM grid data
CMC     
CMC_GEM_reg_.*_.*_([12][0-9][0-9][0-9])([0-1][0-9])([0-3][0-9])([0-2][0-9])_(.*).grib
        FILE    data/pub/native/grid/CMC/GEM/GEM_\1\2\3_\400.grib1
#
# FNMOC data available via IDD (NOGAPS & COAMPS grids)
# -------------------------
FNMOC   ^US058....-.......(0058)_...._.........(....)(..)(..)(..)_(.*)
        FILE    data/pub/native/grid/FNMOC/NOGAPS/NOGAPS_\2\3\4_\5_\1_\6
FNMOC   
^US058....-.......([^0]...|0[^0]..|00[^5].|005[^8])_...._.........(....)(..)(..)(..)_(.*)
        FILE    data/pub/native/grid/FNMOC/COAMPS/COAMPS_\2\3\4_\5_\1_\6
#
# WSETA 
SPARE   ^(20........)/(model_center.dat)
        FILE    -close  data/pub/native/grid/UNIDATA/WSETA/\1_\2
SPARE   ^rad/floater/(..........)/(n0r_........_....)
        FILE    -close  data/pub/native/grid/UNIDATA/WSETA/n0r/\1/WSETA_\2
#
#               
# NPORT
#       
NPORT   ^[LM].B... KWBE ([0-3][0-9])([0-2][0-9])([0-6][0-9])
        FILE    
data/pub/native/grid/NCEP/NAM/CONUS_12km/NAM_CONUS_12km_(\1:yyyy)(\1:mm)\1_\200.grib2
NPORT   ^[LM].S... KWBE ([0-3][0-9])([0-2][0-9])([0-6][0-9])
        FILE    
data/pub/native/grid/NCEP/NAM/Alaska_11km/NAM_Alaska_11km_(\1:yyyy)(\1:mm)\1_\200.grib2
NPORT   ^[LM].E... KWBD ([0-3][0-9])([0-2][0-9])([0-6][0-9])
        FILE    
data/pub/native/grid/NCEP/DGEX/CONUS_12km/DGEX_CONUS_12km_(\1:yyyy)(\1:mm)\1_\200.grib2
NPORT   ^[LM].F... KWBD ([0-3][0-9])([0-2][0-9])([0-6][0-9])
        FILE    
data/pub/native/grid/NCEP/DGEX/Alaska_12km/DGEX_Alaska_12km_(\1:yyyy)(\1:mm)\1_\200.grib2
NPORT   ^[LM].U... KWBN ([0-3][0-9])([0-2][0-9])([0-6][0-9])
        FILE    
data/pub/native/grid/NCEP/NDFD/CONUS_5km/NDFD_CONUS_5km_(\1:yyyy)(\1:mm)\1_\200.grib2
#NPORT  ^[LM].I... KWBJ ([0-3][0-9])([0-2][0-9])([0-6][0-9])
#       FILE    
data/pub/native/grid/NCEP/NCEP/GFS/CONUS_191km/GFS_CONUS_191km_(\1:yyyy)(\1:mm)\1_\200.grib2
#
#
# surface data in netCDF format
#
# metar decoder, creates NetCDF files formated Surface_METAR_yyyymmdd_hh00.nc
# All aviation reports including metar tests. old netcdf file format
DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
        PIPE    decoders/metar2nc
        etc/metar.cdl
        data/pub/decoded/netcdf/surface/metarold
#
# metar decoder, creates NetCDF files with new CDL format on a daily basis
# formated data/decoded/yyyymmdd_metar.nc
DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
        PIPE    decoders/metar2nc.new1
        etc/metar.cdl.new1
        data/pub/decoded/netcdf/surface/metar
#
# upper air perl decoder
DDS|IDS ^U[ABDEFGHIJKLMPQRSTXZ].... .... ([0-3][0-9])
        PIPE    decoders/ua2nc
        etc/raob.cdl
        data/pub/decoded/netcdf/upperair
#
# buoy reports in yymmddhh_buoy.nc decoded into NetCDF format
DDS|IDS ^S[NS].... .... ([0-3][0-9])
        PIPE    decoders/buoy2nc
        e,cdl
        data/pub/decoded/netcdf/surface/buoy
#
# Synoptic reports in yymmddhh_syn.nc decoded into NetCDF format
DDS|IDS ^SM.... .... ([0-3][0-9])
        PIPE    decoders/syn2nc
        etc/syn.cdl
        data/pub/decoded/netcdf/surface/synoptic
#
# grid data in netCDF format
#
# GFS grid on Lambert conformal CONUS grid replacing avn-q
HRS     ^Y.Q... KWB. (..)(..).*/m(AVN|SSIAVN|GFS|SSIGFS)
        PIPE    decoders/gribtonc
        etc/gfs_211.cdl
        
data/pub/decoded/netcdf/grid/NCEP/GFS/CONUS_80km/GFS_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# GFS grid on thinned grids, interpolated to global 5.0 x 2.5 regular grid
HRS     ^H.[I-P]... KWB. (..)(..).*/m(AVN|SSIAVN|GFS|SSIGFS)
        PIPE    decoders/gribtonc
        -q "lin,dlat=2.5,dlon=5.0" etc/gfs_37-44.cdl
        
data/pub/decoded/netcdf/grid/NCEP/GFS/Global_5x2p5deg/GFS_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# ECMWF grid on global 2.5 x 2.5 degree grid
HRS     ^H.[A-L]... ECMF (..)(..)
        PIPE    decoders/gribtonc
        etc/ecmwf_1-12.cdl
        
data/pub/decoded/netcdf/grid/ECMWF/Global_2p5/ECMWF_Global_2p5_(\1:yyyy)(\1:mm)\1_\200.nc
#
# ETA grid on Lambert conformal CONUS grid
HRS     ^[YZ].Q... KWB. (..)(..).*/m(ETA|NAM)
        PIPE    decoders/gribtonc
        etc/eta_211.cdl
        
data/pub/decoded/netcdf/grid/NCEP/NAM/CONUS_80km/NAM_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# GFS grid on global 5.0 x 5.0 degree grid
HRS     ^H.[EF]... KWB. (..)(..)
        PIPE    decoders/gribtonc
        etc/gfs_25-26.cdl
        
data/pub/decoded/netcdf/grid/NCEP/GFS/Extended_Global_5p0deg/GFS_Extended_Global_5p0deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# NGM grid on Lambert conformal CONUS grid
#HRS    ^Y.Q... KWB. (..)(..).*/mNGM
#       PIPE    decoders/gribtonc
#       etc/ngm_211.cdl
#       
data/pub/decoded/netcdf/grid/NCEP/NGM/CONUS_80km/NGM_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# NOW and PWAV grids on global 5.0 x 2.5 degree grids
HRS     ^H.[A-D]... .... (..)(..).*(/mNOW|/mPWAV)
        PIPE    decoders/gribtonc
        etc/ocean_21-24.cdl
        
data/pub/decoded/netcdf/grid/NCEP/OCEAN/Global_5x2p5deg/OCEAN_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# RUC grid on Lambert conformal CONUS grid
HRS     ^[YZ].Q... KWB. (..)(..).*/mRUC
        PIPE    decoders/gribtonc
        etc/ruc_211.cdl
        
data/pub/decoded/netcdf/grid/NCEP/RUC/CONUS_80km/RUC_CONUS_80km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# RUC2 grid on Lambert conformal CONUS grid
HRS     ^Y.W... KWB. (..)(..).*/mRUC2
        PIPE    decoders/gribtonc
        etc/ruc2_236.cdl
        
data/pub/decoded/netcdf/grid/NCEP/RUC2/CONUS_40km/RUC2_CONUS_40km_(\1:yyyy)(\1:mm)\1_\200.nc
#
# SST grid on global 5.0 x 2.5 degree grids
HRS     ^H.[A-D]... .... (..)(..).*/mSST
        PIPE    decoders/gribtonc
        etc/sst_21-24.cdl
        
data/pub/decoded/netcdf/grid/NCEP/SST/Global_5x2p5deg/SST_Global_5x2p5deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# SST grid on global 2.0 x 2.0 degree grid
HRS     ^H.[T-W]... .... (..)(..).*/mSST
        PIPE    decoders/gribtonc
        etc/sst_61-64.cdl
        
data/pub/decoded/netcdf/grid/NCEP/SST/Global_2x2deg/SST_Global_2x2deg_(\1:yyyy)(\1:mm)\1_\200.nc
#
# png compressed NEXRAD Level III radar composites in GINI format
#FNEXRAD                ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
#               FILE
#               
data/pub/native/radar/composite/\2/\1/\3/Level3_Composite_\2_\1_\3_\4.gini
#
#
# NOAAPORT GINI Images
#
# png compressed NEXRAD Level III radar composites in GINI format
FNEXRAD ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
        PIPE    -close
        util/pngg2gini -vl logs/ldm-mcidas.log
        
data/pub/native/radar/composite/\2/\1/\3/Level3_Composite_\2_\1_\3_\4.gini
#
# end of pqact.thredds
#
#
#PCWS   ^FSL\.NetCDF\.ACARS\.QC\.(...........)
#       FILE    -close  data/ucar_only/decoded/netcdf/acars/\1/\1.nc
# CONDUIT data directory structure
#
# pub
# pub/native
# pub/native/grid
# pub/native/grid/NCEP
#
# pub/native/grid/NCEP/GFS/Global_0p5deg
# pub/native/grid/NCEP/GFS/Global_onedeg
# pub/native/grid/NCEP/GFS/Global_2p5deg
#
# pub/native/grid/NCEP/RUC2/CONUS_20km/hybrid
# pub/native/grid/NCEP/RUC2/CONUS_20km/pressure
# pub/native/grid/NCEP/RUC2/CONUS_20km/surface
#
# pub/native/grid/NCEP/NAM/Alaska_45km/conduit
# pub/native/grid/NCEP/NAM/CONUS_20km/selectsurface
# pub/native/grid/NCEP/NAM/CONUS_20km/surface
# pub/native/grid/NCEP/NAM/CONUS_40km/conduit
# pub/native/grid/NCEP/NAM/Polar_90km
#
# pub/native/grid/NCEP/GFS/Global_1p0deg_Ensemble
# pub/native/grid/NCEP/GFS/Global_2p5deg_Ensemble
#
# CONDUIT Data actions
#
# GFS grids
# example product
#/afs/.nwstg.nws.noaa.gov/ftp/SL.us008001/ST.opnl/MT.gfs_CY.12/RD.20041213/PT.grid_DF.gr1/fh.0078_tl.press_gr.onedeg
 
#!grib2/ncep/AVN/#000/200412131200/F6900/RELH/3300-10000 SGMA!
#
# GFS grids Global_0p5, Global_onedeg, and Global_2p5 grids
CONDUIT         
MT.gfs_CY.(..)/RD.(........)/PT.grid_DF.gr([12])/(fh.....)_tl.press_gr.(......)
        FILE    -close  
data/pub/native/grid/NCEP/GFS/Global_\5/GFS_Global_\5_\2_\100.grib\3
#
# RUC2 grids
#
# hybrid
CONDUIT         
MT.ruc_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.....)_tl.press_gr.bgrib20
        FILE    -close  
data/pub/native/grid/NCEP/RUC2/CONUS_20km/hybrid/RUC2_CONUS_20km_hybrid_\2_\100.grib1
#
# surface
CONDUIT         
MT.ruc_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.....)_tl.press_gr.sgrib20
        FILE    -close  
data/pub/native/grid/NCEP/RUC2/CONUS_20km/surface/RUC2_CONUS_20km_surface_\2_\100.grib1
#
# pressure
CONDUIT         
MT.ruc_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.....)_tl.press_gr.us20km
        FILE    -close  
data/pub/native/grid/NCEP/RUC2/CONUS_20km/pressure/RUC2_CONUS_20km_pressure_\2_\100.grib1
#
# NAM grids
#
# example product
#/afs/.nwstg.nws.noaa.gov/ftp/SL.us008001/ST.opnl/MT.nam_CY.12/RD.20050813/PT.grid_DF.gr1/fh.0027_tl.press_gr.awipak
 !grib/ncep/ETA_84/#216/200508131200/F027/HGT/sfc! 000009
#
# Alaska_45km
CONDUIT         
MT.nam_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.....)_tl.press_gr.(......) 
!grib/ncep/ETA_84/#216
        FILE    -close  
data/pub/native/grid/NCEP/NAM/Alaska_45km/conduit/NAM_Alaska_45km_conduit_\2_\100.grib1
#
# selectsurface
CONDUIT         
MT.nam_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.....)_tl.press_gr.icwf20 
!grib/ncep/ETA_84/#215
        FILE    -close  
data/pub/native/grid/NCEP/NAM/CONUS_20km/selectsurface/NAM_CONUS_20km_selectsurface_\2_\100.grib1
#
# surface
CONDUIT         
MT.nam_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.....)_tl.press_gr.awip20 
!grib/ncep/ETA_84/#215
        FILE    -close  
data/pub/native/grid/NCEP/NAM/CONUS_20km/surface/NAM_CONUS_20km_surface_\2_\100.grib1
#
# conduit feed
CONDUIT         
MT.nam_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.....)_tl.press_gr.(......) 
!grib/ncep/ETA_84/#212
        FILE    -close  
data/pub/native/grid/NCEP/NAM/CONUS_40km/conduit/NAM_CONUS_40km_conduit_\2_\100.grib1
#
# Polar_90km
CONDUIT         
MT.nam_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.....)_tl.press_gr.(......) 
!grib/ncep/ETA_84/#104
        FILE    -close  
data/pub/native/grid/NCEP/NAM/Polar_90km/NAM_Polar_90km_\2_\100.grib1
#
# MRF Ensemble grids
#
# example products
#/afs/.nwstg.nws.noaa.gov/ftp/SL.us008001/ST.opnl/MT.ensg_CY.12/RD.20050812/PT.grid_DF.gr1/fh.xxxx_tl.press_pa.z1000
 !grib/ncep/AVN/#002/200508121200/F300/HGT/1000_mb! 000040
#  
/afs/.nwstg.nws.noaa.gov/ftp/SL.us008001/ST.opnl/MT.ensg_CY.12/RD.20050812/PT.grid_DF.gr1/fh.xxxx_tl.press_pa.z1000
 !grib/ncep/SPEC_62_MRF/#002/200508121200/F006/HGT/1000_mb! 000049
#
# Global_1p0deg_Ensemble
CONDUIT         
MT.ensg_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.xxxxhr)_tl.press_pa.(.*) !grib
        FILE    -close  
data/pub/native/grid/NCEP/GFS/Global_1p0deg_Ensemble/GFS_Global_1p0deg_Ensemble_\2_\100.grib1
#
# Global_2p5deg_Ensemble
CONDUIT         
MT.ensg_CY.(..)/RD.(........)/PT.grid_DF.gr1/(fh.xxxx)_tl.press_pa.(.*) !grib
        FILE    -close  
data/pub/native/grid/NCEP/GFS/Global_2p5deg_Ensemble/GFS_Global_2p5deg_Ensemble_\2_\100.grib1
#
#
#NMC2|SPARE     ^.status\.(.*) [0-9][0-9][0-9][0-9][0-9][0-9]
#       FILE    -close  data/pub/native/other/conduit/status/\1
#
# Soundings Products
#
# GFS soundings
#CONDUIT                
MT.gfs_CY.(..)/RD.(........)/PT.sndn_DF.bufr/(fh.....)_tl.press_gr.(......)
#       FILE    -close  
data/pub/native/grid/NCEP/GFS/Global_\5/GFS_Global_\5_\2_\100.grib\3
#
# This pqact defines the directory structure expected by the THREDDS Data Server
# default configuration.  It includes the pqact entries using decoders and FILE
# actions to create the structure. If this pqact is implemented then the 
# THREDDS Data Server should work without doing any data structure 
# configurations. Default configuration starting at the ~ldm/data directory:
#
# pub
# pub/native
# pub/native/radar/level2/{station}/{day}
# pub/native/radar/level3/{day}/{station}/{product}
#
#
# level2 radar from CRAFT  feed
#
CRAFT   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9])
        FILE    data/pub/native/radar/level2/\1/\2/Level2_\1_\2_\3.ar2v
#
# level3 radar from NEXRAD feedtype
#
NNEXRAD ^SDUS[2357]. .... ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
        FILE    -close  
data/pub/native/radar/level3/\4/\5/(\1:yyyy)(\1:mm)\1/Level3_\5_\4_(\1:yyyy)(\1:mm)\1_\2\3.nids
#
#
# Configuration file for "manageFilesDirs.pl" utility.
#
# This utility manages the number of files in a directory structure.
#
# @param  startdir the directory where to start managing
# @param  pattern  the pattern to match files or directories, yyyymmdd is a 
#         place holder for year, month, day directories
# @param  number  the number of days of data to keep
#
# Once the script matches the pattern, it deletes the files or directories and
# then returns to the directory up one level. It skips files or directories
# that are links and prints out a message on files or directories that no
# action was taken on it. 
#
# Each line consists of a directory, a pattern, and the number of days to 
retain.
#
# Directory                                             pattern         days 
#
/data/ldm/pub/native/surface/metar                      yyyymmdd        7
/data/ldm/pub/native/surface/land                       bufr            7
/data/ldm/pub/native/surface/satellite                  bufr            7
/data/ldm/pub/native/surface/satellite1                 bufr            7
/data/ldm/pub/native/surface/satellite2                 bufr            7
/data/ldm/pub/native/surface/satellite3                 bufr            7
/data/ldm/pub/native/soundings                          bufr            7
/data/ldm/pub/native/synoptic                           bufr            7
/data/ldm/pub/native/profiler/bufr                      bufr            7
/data/ldm/pub/native/satellite                          yyyymmdd        7
/data/ldm/pub/native/radar/level2                       yyyymmdd        7
/data/ldm/pub/native/radar/level3                       yyyymmdd        7
/data/ldm/pub/native/radar/level3/DPA                   dpa             7
/data/ldm/pub/native/radar/composite                    yyyymmdd        7
/data/ldm/pub/native/radar/10km_mosaic                  yyyymmdd        7
/data/ldm/pub/native/profiler/RASS                      yyyymmdd        7
/data/ldm/pub/native/profiler/wind                      yyyymmdd        7
/data/ldm/pub/native/nwstg/text                         yyyymmdd        7
/data/ldm/pub/native/other/hrs                          HRS.wmo         7
/data/ldm/pub/native/other                              bufr            7
/data/ldm/pub/native/conduit                            RD.20           2
/data/ldm/pub/native/grid/CMC                           grib.$          7
/data/ldm/pub/native/grid/ECMWF                         grib.$          7
/data/ldm/pub/native/grid/NCEP                          grib.$          7
/data/ldm/pub/native/grid/UKMET                         grib.$          7
/data/ldm/pub/native/grid/UNIDATA/WSETA/n0r             yyyymmdd        7
#
/data/ldm/pub/decoded/netcdf/surface                    Surface         7
/data/ldm/pub/decoded/netcdf/upperair                   Upperair        7
/data/ldm/pub/decoded/netcdf/grid                       nc$             7
#
#!/usr/bin/perl
#
# Name:         manageFileDirs.pl
# 
# Author:       Robb Kambic
# Date  :       Sept 2, 2005
# 
# Purpose:      manage files in a directory sturcture.
#    
# Description:  
#
# @param  startdir the directory where to start managing
# @param  pattern  the pattern to match files or directories, yyyymmdd is a 
#         place holder for year, month, day directories
# @param  number  the number of days to keep
#
# @param  configuration file that has the above parameters on one line.
#
# Once the script matches the pattern, it deletes the files or directories and
# then returns to the directory up one level. It skips files or directories
# that are links and prints out a message on files or directories that no
# action was taken on it. 
#
# process command line switches

while ($_ = $ARGV[0], /^-/) {
         shift;
       last if /^--$/;
              /^-D(.*)/ && ($debug = $1);
             /^(-v)/ && $verbose++;
             /^(-d)/ && ( $startDir = shift ) ;
             /^(-p)/ && ( $pattern = shift ) ;
             /^(-n)/ && ( $number = shift ) ;
             /^(-f)/ && ( $conf = shift ) ;
}
# configuration file given to process
if( defined( $conf ) ) {
        print "Start ", `/bin/date`;
        open( CONF, "$conf" ) || die "cannot open $conf $!";
        while( <CONF> ) {
                next if( /^#/ );
                chop();
                ( $startDir, $pattern, $number ) = split( /\s+/, $_ );
                next if( $pattern eq "" || $number eq "" );
                startChecking();
        }
        close CONF;
        print "End ", `/bin/date`;
} else {
        exit 1 if( $pattern eq "" || $number eq "" );
        print "Start ", `/bin/date`;
        startChecking();
        print "End ", `/bin/date`;
}

sub startChecking {

if( ! -d $startDir ) {
        print "$startDir doesn't exist\n";
        return;
}
chdir( $startDir );
if( $pattern =~ /yyyymmdd/i ) {
        $pattern = "^\\d{8}"; # pattern set to 8 digits
}
opendir( TOP, $startDir ) || die "cannot open $startDir $!";
( @TOP ) = readdir( TOP );
closedir( TOP );
checkDirs( @TOP );

} # end startChecking

# checkDirs is a recursive routine used to walk the directory structure in a
# depth first search for the designated pattern. Once the pattern is found, it 
# removes the files/directories in excess of the days to keep.  After deleting
# files or directories matching the pattern, the routine returns up one 
# directory level.
#
sub checkDirs{

my ( @INODES ) = @_;
local( $i, $j, $delete, @subINODES );

#print "currently in ", `pwd`;
for( $i = 0; $i <= $#INODES; $i++ ) {
        # skip ., .., and links
        next if( $INODES[ $i ] =~ /^\.$|^\.\.$/ || -l $INODES[ $i ] );
        if( $INODES[ $i ] =~ /$pattern/ ) {
                print `/bin/date`;
                print "currently in ", `pwd`;
                # just manage files/dirs matching pattern
                $delete = "";
                for( $j = 0; $j <= $#INODES; $j++ ) {
                        next if( $INODES[ $j ] =~ /^\.$|^\.\.$/ || -l $INODES[ 
$j ] );
                        next unless ( $INODES[ $j ] =~ /$pattern/ && -M 
$INODES[ $j ] > $number );
                        $delete = $delete . " $INODES[ $j ]";
                        # if grib file delete gbx index and inventory too
                        if( $INODES[ $j ] =~ /grib.$/ ) {
                                $delete = $delete . " $INODES[ $j ]" . ".gbx";
                                $delete = $delete . " $INODES[ $j ]" . 
".fmrInv.xml";
                        }
                        # if bufr file delete bfx index too
                        if( $INODES[ $j ] =~ /bufr$/ ) {
                                $delete = $delete . " $INODES[ $j ]" . ".bfx";
                        }
                } 
                if( $delete ne "" ) {
                        print "`/bin/rm -r $delete`\n";
                        `/bin/rm -r $delete`;
                }
                #print "currently in ", `pwd`;
                return;
        } elsif( -d $INODES[ $i ] ) {
                opendir( DIR, $INODES[ $i ] ) || 
                        die "cannot open $INODES[ $i ] $!";
                ( @subINODES ) = readdir( DIR );
                closedir( DIR );
                chdir( $INODES[ $i ] );
                #print "currently in ", `pwd`;
                checkDirs( @subINODES );
                chdir( ".." );
                #print "currently in ", `pwd`;
        } elsif( $INODES[ $i ] =~ /\.scour/ ) {
                print "`/bin/rm $INODES[ $i ]`\n";
                `/bin/rm $INODES[ $i ]`;
        } else {
                print "currently in ", `pwd`;
                print "No action taken on $INODES[ $i ]\n";
        } 
}
} # end checkDirs