[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #VHO-548969]: question about pipes and files


> My name is Jon Wolfe and I work for a wind-forecasting company.  We use
> LDM to download radar data and other data feeds, but none of us is
> an expert with LDM.  The issue we're seeing is that we have a chain
> of commands in pqact we're trying to understand better.  Basically,
> as radar files come in we're trying to use a pipe command to convert
> them and exec to run ready_radar:
> NEXRD2  
> ^L2-([^/]*)/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9][0-9][0-9])
> PIPE    -close  nexradII /lustrefs/wind/data/radar/work/\2/\3
> NEXRD2  
> ^L2-([^/]*)/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9][0-9][0-9])
> EXEC    bin/ready_radar.sh \2 \3 /lustrefs/wind/data/radar/work 
> /lustrefs/wind/data/radar/netCDF
> However, we're seeing our system get hammered in that ready_radar runs
> over and over again as the files come in.  So my question is, how can we
> get the EXEC command to wait until the file has been completely downloaded
> before running?  The end result of all the hammering of the system has
> been that some of the radar files don't ever come in completely and the
> partial files cause us all kinds of problems.

The two regular expressions look identical, so the associated actions will 
execute on the same data-products. This likely means that "ready_radar.sh" will 
be executed every time a radar product arrives -- regardless of whether or not 
the volume-scan to which it belongs is complete.

I think the last product of a volume-scan has a "\E" suffix (you could check 
with notifyme(1)). If there's more than one path for radar products to get to 
your system, however, then that product can arrive before the volume-scan is 

I'd be inclined to combine the "nexradII" and "ready_radar.sh" programs into a 
single script (whose first significant action would be to execute "nexradII"). 
The rest of the script would determine if the volume-scan is complete and 
execute the equivalent of "ready_radar.sh" if it was.

I've BCC'ed our expert here on NEXRAD Level II processing in case they have 
something to add.

> Thanks for your help,
> Jon

Steve Emmerson

Ticket Details
Ticket ID: VHO-548969
Department: Support LDM
Priority: Normal
Status: Closed

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.