[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #BJE-199821]: radar data

Hi Elmer,

> Thanks ... and since I am totally unsure of how this is set up, I'm
> fairly sure the request should be something like
> request FNEXRAD "^SDUS3(.*)" idd.cise-nsf.gov
> at least I think the table says the 1km BREF is SDUS,and think the line
> in pqact.conf should be something like
> FNEXRAD ^SDUS3(..) (....) ([0-3][0-9])([0-2][0-9])([0-5][0-9])
> FILE    -close  /data/radar/\1
> Is this anywhere near correct?

Your instincts were good, but the headers for the FNEXRAD products are
created by us, not the NWS, so your request pattern would not work.

The easiest/best way to figure out what kind of pattern you need/want
in ldmd.conf request lines is by examining the output of the LDM utility
'notifyme' which will include the headers of products.  For instance,

<as 'ldm'>
notifyme -vxl- -ff FNEXRAD -h idd.cise-nsf.gov -o 3600

will result in a listing that looks like:

Jul 25 14:48:18 notifyme[29605] NOTE: Starting Up: idd.cise-nsf.gov: 
20070725134818.304 TS_ENDT {{FNEXRAD,  ".*"}}
Jul 25 14:48:18 notifyme[29605] NOTE: LDM-5 desired product-class: 
20070725134818.304 TS_ENDT {{FNEXRAD,  ".*"}}
Jul 25 14:48:18 notifyme[29605] INFO: Resolving idd.cise-nsf.gov to took 0.00054 seconds
Jul 25 14:48:18 DEBUG: NOTIFYME(idd.cise-nsf.gov) returns OK
Jul 25 14:48:18 notifyme[29605] NOTE: NOTIFYME(idd.cise-nsf.gov): OK
Jul 25 14:48:19 notifyme[29605] INFO: 386d1a6b45b100e60653b41e6e1bb289   306362 
20070725134828.723 FNEXRAD 000  radar_mosaic_national 
!grib/unidata/UPC_1/#255/200707251343/F000/N0R/sfc! 000000
Jul 25 14:48:19 notifyme[29605] INFO: 05a9b49fe0f128abb2aa8e1e4da71908   270252 
20070725134828.733 FNEXRAD 001  radar_mosaic_national 
!grib/unidata/UPC_1/#255/200707251343/F000/NET/sfc! 000001
Jul 25 14:48:19 notifyme[29605] INFO: 302d3ecebbf97387d5066d99cb663a60  3621501 
20070725134828.771 FNEXRAD 000  radar_mosaic_nathr 
!grib2/unidata/UPC_255/#000/200707251343F000/N0R/0 - NONE! 000000
Jul 25 14:48:19 notifyme[29605] INFO: 8e8e2405a15bc92dc00322f0d4c5f003    72284 
20070725134830.920 FNEXRAD 000  pnga2area Q5 RL 300 RADAR BREF 6km 20070725 1343
Jul 25 14:48:20 notifyme[29605] INFO: 78b14c9dab2fcac2d58aa1e8a15f6d3b   125633 
20070725134901.176 FNEXRAD 000  rad/NEXRCOMP/2km/n1p_20070725_1347
Jul 25 14:48:20 notifyme[29605] INFO: 4f650380a2b4a3fda84c7179a17a8907  1696321 
20070725135000.999 FNEXRAD 000  rad/NEXRCOMP/1km/n0r_20070725_1346
Jul 25 14:48:20 notifyme[29605] INFO: f99a603f0aebb5ba8424eef2d6bcf716   137082 
20070725135001.071 FNEXRAD 000  rad/NEXRCOMP/4km/ntp_20070725_1348
Jul 25 14:48:20 notifyme[29605] INFO: 3b1766ab58335db57b7a5bd2298c2284   306362 
20070725135222.139 FNEXRAD 000  radar_mosaic_national 
!grib/unidata/UPC_1/#255/200707251348/F000/N0R/sfc! 000000
Jul 25 14:48:20 notifyme[29605] INFO: 9bc4cf0e608fce3c985dd8d50925f313  3649622 
20070725135222.196 FNEXRAD 000  radar_mosaic_nathr 
!grib2/unidata/UPC_255/#000/200707251348F000/N0R/0 - NONE! 000000

This 'notifyme' invocation asks the upstream host ('-h idd.cise-nsf.gov') to 
list all
of the FNEXRAD ('-f FNEXRAD') products it has received in the last hour ('-o 
(NB: if you leave off the host specification, 'notifyme' will list the products 
your local LDM is receiving.)

The product headers in this listing snippit are:

radar_mosaic_national !grib/unidata/UPC_1/#255/200707251343/F000/N0R/sfc! 000000
radar_mosaic_national !grib/unidata/UPC_1/#255/200707251343/F000/NET/sfc! 000001
radar_mosaic_nathr !grib2/unidata/UPC_255/#000/200707251343F000/N0R/0 - NONE! 
pnga2area Q5 RL 300 RADAR BREF 6km 20070725 1343
radar_mosaic_national !grib/unidata/UPC_1/#255/200707251348/F000/N0R/sfc! 000000
radar_mosaic_nathr !grib2/unidata/UPC_255/#000/200707251348F000/N0R/0 - NONE! 

This abbreviated listing (the 'notifyme' invocation will continue to run showing
you products as they are received on the upstream host) shows that there are 
different kinds of products in the FNEXRAD datastream:

- grib messages (the products that contain '!grib/unidata/UPC_' in them

- images in McIDAS AREA format that have been PNG compressed (products that 
  'pnga2area' in the header)

- images in PNG-compressed GINI format (products that have 'rad/NEXRCOMP' in the

Since you are interested in the 1 km national N0R (aka BREF) composite, you can 
a pattern like:


in your ldmd.conf request and pqact.conf pattern-action entries.

Example: the following ldmd.conf entry would request only the 1km N0R
composite images:

request FNEXRAD ^rad/NEXRCOMP/1km idd.cise-nsf.gov

Example: the following pqact.conf entry would file all of the composites into
a hierarchical directory structure under ~ldm/data:

# PNG-compressed GINI NEXRAD Level III composite images
FNEXRAD ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
        FILE    -close  data/images/NEXRCOMP/\1/\2/\2_\3_\4

Now a word of caution:

The format of entries in LDM pattern-action files (e.g., pqact.conf) is
very rigid.  Some of the whitespace _MUST_ be tabs, not spaces.  The example
entry I show above is read as:


Also, pattern-action file entries do not allow any lines that begin with
spaces.  Actions in the file that are located after a line that begins
with one or more spaces will _not_ be executed.

When you create or modify an action in a pattern-action file,
you should run 'ldmadmin pqactcheck' to verify that the syntax of the
entry is correct (syntax only, not content):

cd ~ldm
ldmadmin pqactcheck                     <- checks entries in etc/pqact.conf
ldmadmin pqactcheck -p etc/pqact.local  <- checks entries in etc/pqact.local

Finally, as indicated in the comment line for the pqact.conf entry above, the
FNEXRAD images are in a UPC-developed PNG-compressed format (the headers of
products are not compressed, but the image portion is).  This means that you
will likely need to uncompress the images before you use them (unless you
use one of the Unidata-developed display/analysis applications that support
the PNG-compressed format directory (e.g., Unidata GEMPAK and Unidat McIDAS).
We provide a package of 'decoders', ldm-mcidas, that can be used to do the
uncompression at sites that want to use the image products in applications
that do not support the PNG-compressed formats.  We can provide more information
on access to this package if you need it and after you have gotten your data
ingestion working.


Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
Unidata HomePage                       http://www.unidata.ucar.edu

Ticket Details
Ticket ID: BJE-199821
Department: Support IDD
Priority: Normal
Status: Closed

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.