[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #CIX-410192]: GOES R script

Hi Gerry,

re:
> Saw this after I'd left the office and was driving to Texas on Friday;
> thank you.

No worries.

re:
> Now a question . Reading the README.md file, it says Python3 is
> required?

Yes, Python 2.x will _not_ work.

re:
> Most distro's don't natively come with 3, but I suspect I can
> make it work... I'll give this a try in the next 10 min.

We installed python as part of Miniconda3.  After this is installed,
we added netCDF to the list of things that it contains so that we
easily have access to a pre-built netCDF4.

> address@hidden> wrote:
> 
> > Hi Gerry,
> >
> > re: R99 products are NOT coming in on NOAAPort
> >
> > > Sounds good. Explains why I'm not seeing them!
> >
> > Yup :-)
> >
> > re: PUG
> >
> > > Thanks. Snagged.
> >
> > re: relative directory/file reference in action not being found
> >
> > > And... I got them working. THanks.
> > re: kick the tires on the NIMAGE datastream revamp?
> >
> > > Yes, please. I'd like to look at the new data.
> >
> > OK.  What is the fully qualified name and IP address of the
> > machine you intend to use to REQUEST the revamped NIMAGE feed?
> >
> > re:
> > > AND, on another note...
> > >
> > > Recalling the old pqact.conf and scripting to handle full volume scans
> > for
> > > GEMPAK, until today I've been getting and aggregating radial scans, and
> > > letting them fill up. We've changed our processing lately, and PyART
> > wants
> > > full volumes, apparently (I need to independently confirm that...) so I
> > > need to go back to making those files visible when they're complete
> > (i.e.,
> > > the /E flag is set). I've got the following (please assume all tabs are
> > > correct):
> > > NEXRAD2
> > >
> > ^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9])/([0-9]*)/
> > > FILE    /work/LDM/NEXRAD2/\1/.\1_\2_\3
> > >
> > > NEXRAD
> > >
> > ^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9]).*/E/
> > > EXEC    scripts/ProcessRadar.csh /work/LDM/NEXRAD2/\1/.\1_\2_\3
> > /work/LDM/NEXRAD2/\1/\1_\2_\3
> > >
> > > and ProcessRadar.csh looks like:
> > > #!/bin/csh -V
> > > #
> > > ## Shell script to move a file in the data tree, with check
> > > # to avoid overwriting data (in case of retransmission on data).
> > >
> > > if ($#argv != 2 ) then
> > > echo "Usage: $0 tmpfile finalfile"
> > > exit 0
> > > endif
> > >
> > > echo `date -u` "tmpFile: $1; finalFile: $2" >>
> > /usr/local/ldm/var/logs/NEXRAD2_error.log
> > >
> > > if ( ! -e $1 ) then
> > > echo "could not find $1" >>! /usr/local/ldm/var/logs/NEXRAD2_error.log
> > > exit 0
> > > endif
> > >
> > > if ( ! -e $2 ) then
> > > mv $1 $2
> > > else
> > > # don't overwrite outfile!
> > > # Use process ID for unique file name to save old file
> > > # Useful if more than 1 scan volume this minute
> > > mv $2 $2.$$
> > > mv $1 $2
> > > echo `date` "mv $1 $2" >>! /usr/local/ldm/var/logs/NEXRAD2_error.log
> > > endif
> > > exit 0
> > >
> > > I'm getting child errors with a less than useful error code:
> > > 20190419T184935.562197Z                pqact[17673]
> > > filel.c:reap() ERROR Child 23949 exited with status 1
> > >
> > > Any suggestions? I suspect it's a silly error...
> >
> > A couple of comments:
> >
> > - there is probably a typo or something similar in the script
> >
> > - stitching together the NEXRAD Level 2 chunks distributed in the IDD
> >   NEXRAD2 feed is much more difficult than the simple approach
> >   that Chiz implemented and let GEMPAK users know about a long time
> >   ago
> >
> >   Why?
> >
> >   Because, you can _not_ count on the pieces of a volume scan being
> >   received in order.  This means that you can get a NEXRAD2 product that
> >   is marked as the last piece of a volume scan before you get other pieces
> >   If your processing assumes that the product with the '/E' really is the
> >   last piece of a volume scan, then you will end up with files that are
> >   incomplete.
> >
> >   What to do?
> >
> >   We originally promoted the use of a Perl script that Robb Kambic
> > developed
> >   for doing the restitch, but we have more recently (well over a year ago)
> >   moved to a Python script that does a better job.
> >
> > The Python script that I am referring to is named 'l2assemble.py' and it
> > is part of the ldm-alchemy package that can be found in the Unidata
> > collection on GitHub:
> >
> > https://github.com/Unidata/ldm-alchemy
> >
> > As an FYI:  here is how we run 'l2assemble.py' here in the UPC:
> >
> > #
> > # ---------------------------------
> > # - NEXRAD Level II Radar WSR-88D -
> > # ---------------------------------
> > #
> > CRAFT
> >  
> > ^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/([0-9]*)/[0-9]*/(S|I|E)
> >         PIPE    -metadata
> >         etc/TDS/ldm-alchemy/l2assemble.py -g -t 960 -d
> > /data/ldm/pub/native/radar/level2
> >         -f raw --threads 20
> > #
> > # File Model Data Message packets separately, since they are on a different
> > # time schedule
> > #
> > CRAFT
> >  
> > ^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/([0-9]*)/M
> >         FILE    -close
> > /data/ldm/pub/native/radar/level2_model/\1/\2/Level2MDM_\1_\2_\3.ar2v
> >
> > Of course, you will need to correctly reference where you decide to put
> > 'l2assemble.py'
> > on your machine (suggestion:  ~ldm/util), and adjust output directories.
> >
> > NB: I simply clipped out the actions from one of our pattern-action files
> > to which NEXRAD2 is sent for processing.  These actions still use the old
> > name for the NEXRAD Level 2 feed, CRAFT.  I suggest changing these to
> > NEXRAD2.
> >
> > Cheers,
> >
> > Tom
> > --
> >
> > ****************************************************************************
> > Unidata User Support                                    UCAR Unidata
> > Program
> > (303) 497-8642                                                 P.O. Box
> > 3000
> > address@hidden                                   Boulder, CO
> > 80307
> >
> > ----------------------------------------------------------------------------
> > Unidata HomePage                       http://www.unidata.ucar.edu
> >
> > ****************************************************************************
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: CIX-410192
> > Department: Support IDD
> > Priority: Normal
> > Status: Closed
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the
> > Unidata inquiry tracking system and then made publicly available through
> > the web.  If you do not want to have your interactions made available in
> > this way, you must let us know in each email you send to us.
> >
> >
> >
> 
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371
> ++++++++++++++++++++++
> *The way to get started is to quit talking and begin doing.*
> *   Walt Disney*
> 
> 

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CIX-410192
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.