[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #CIX-410192]: GOES R script



Hi Gerry,

re: R99 products are NOT coming in on NOAAPort

> Sounds good. Explains why I'm not seeing them!

Yup :-)

re: PUG

> Thanks. Snagged.

re: relative directory/file reference in action not being found

> And... I got them working. THanks.
re: kick the tires on the NIMAGE datastream revamp?

> Yes, please. I'd like to look at the new data.

OK.  What is the fully qualified name and IP address of the
machine you intend to use to REQUEST the revamped NIMAGE feed?

re:
> AND, on another note...
> 
> Recalling the old pqact.conf and scripting to handle full volume scans for
> GEMPAK, until today I've been getting and aggregating radial scans, and
> letting them fill up. We've changed our processing lately, and PyART wants
> full volumes, apparently (I need to independently confirm that...) so I
> need to go back to making those files visible when they're complete (i.e.,
> the /E flag is set). I've got the following (please assume all tabs are
> correct):
> NEXRAD2
> ^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9])/([0-9]*)/
> FILE    /work/LDM/NEXRAD2/\1/.\1_\2_\3
> 
> NEXRAD
> ^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9]).*/E/
> EXEC    scripts/ProcessRadar.csh /work/LDM/NEXRAD2/\1/.\1_\2_\3 
> /work/LDM/NEXRAD2/\1/\1_\2_\3
> 
> and ProcessRadar.csh looks like:
> #!/bin/csh -V
> #
> ## Shell script to move a file in the data tree, with check
> # to avoid overwriting data (in case of retransmission on data).
> 
> if ($#argv != 2 ) then
> echo "Usage: $0 tmpfile finalfile"
> exit 0
> endif
> 
> echo `date -u` "tmpFile: $1; finalFile: $2" >> 
> /usr/local/ldm/var/logs/NEXRAD2_error.log
> 
> if ( ! -e $1 ) then
> echo "could not find $1" >>! /usr/local/ldm/var/logs/NEXRAD2_error.log
> exit 0
> endif
> 
> if ( ! -e $2 ) then
> mv $1 $2
> else
> # don't overwrite outfile!
> # Use process ID for unique file name to save old file
> # Useful if more than 1 scan volume this minute
> mv $2 $2.$$
> mv $1 $2
> echo `date` "mv $1 $2" >>! /usr/local/ldm/var/logs/NEXRAD2_error.log
> endif
> exit 0
> 
> I'm getting child errors with a less than useful error code:
> 20190419T184935.562197Z                pqact[17673]
> filel.c:reap() ERROR Child 23949 exited with status 1
> 
> Any suggestions? I suspect it's a silly error...

A couple of comments:

- there is probably a typo or something similar in the script

- stitching together the NEXRAD Level 2 chunks distributed in the IDD
  NEXRAD2 feed is much more difficult than the simple approach
  that Chiz implemented and let GEMPAK users know about a long time
  ago

  Why?

  Because, you can _not_ count on the pieces of a volume scan being
  received in order.  This means that you can get a NEXRAD2 product that
  is marked as the last piece of a volume scan before you get other pieces
  If your processing assumes that the product with the '/E' really is the
  last piece of a volume scan, then you will end up with files that are
  incomplete.

  What to do?

  We originally promoted the use of a Perl script that Robb Kambic developed
  for doing the restitch, but we have more recently (well over a year ago)
  moved to a Python script that does a better job.

The Python script that I am referring to is named 'l2assemble.py' and it
is part of the ldm-alchemy package that can be found in the Unidata
collection on GitHub:

https://github.com/Unidata/ldm-alchemy

As an FYI:  here is how we run 'l2assemble.py' here in the UPC:

#
# ---------------------------------
# - NEXRAD Level II Radar WSR-88D -
# ---------------------------------
#
CRAFT   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/([0-9]*)/[0-9]*/(S|I|E)
        PIPE    -metadata
        etc/TDS/ldm-alchemy/l2assemble.py -g -t 960 -d 
/data/ldm/pub/native/radar/level2
        -f raw --threads 20
#
# File Model Data Message packets separately, since they are on a different
# time schedule
#
CRAFT   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/([0-9]*)/M
        FILE    -close  
/data/ldm/pub/native/radar/level2_model/\1/\2/Level2MDM_\1_\2_\3.ar2v

Of course, you will need to correctly reference where you decide to put 
'l2assemble.py'
on your machine (suggestion:  ~ldm/util), and adjust output directories.

NB: I simply clipped out the actions from one of our pattern-action files
to which NEXRAD2 is sent for processing.  These actions still use the old
name for the NEXRAD Level 2 feed, CRAFT.  I suggest changing these to
NEXRAD2.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CIX-410192
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.