[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #VHO-548969]: question about pipes and files

Jon,

> My name is Jon Wolfe and I work for a wind-forecasting company.  We use
> LDM to download radar data and other data feeds, but none of us is
> an expert with LDM.  The issue we're seeing is that we have a chain
> of commands in pqact we're trying to understand better.  Basically,
> as radar files come in we're trying to use a pipe command to convert
> them and exec to run ready_radar:
> 
> NEXRD2  
> ^L2-([^/]*)/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9][0-9][0-9])
> PIPE    -close  nexradII /lustrefs/wind/data/radar/work/\2/\3
> 
> NEXRD2  
> ^L2-([^/]*)/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9][0-9][0-9])
> EXEC    bin/ready_radar.sh \2 \3 /lustrefs/wind/data/radar/work 
> /lustrefs/wind/data/radar/netCDF
> 
> However, we're seeing our system get hammered in that ready_radar runs
> over and over again as the files come in.  So my question is, how can we
> get the EXEC command to wait until the file has been completely downloaded
> before running?  The end result of all the hammering of the system has
> been that some of the radar files don't ever come in completely and the
> partial files cause us all kinds of problems.

The two regular expressions look identical, so the associated actions will 
execute on the same data-products. This likely means that "ready_radar.sh" will 
be executed every time a radar product arrives -- regardless of whether or not 
the volume-scan to which it belongs is complete.

I think the last product of a volume-scan has a "\E" suffix (you could check 
with notifyme(1)). If there's more than one path for radar products to get to 
your system, however, then that product can arrive before the volume-scan is 
complete.

I'd be inclined to combine the "nexradII" and "ready_radar.sh" programs into a 
single script (whose first significant action would be to execute "nexradII"). 
The rest of the script would determine if the volume-scan is complete and 
execute the equivalent of "ready_radar.sh" if it was.

I've BCC'ed our expert here on NEXRAD Level II processing in case they have 
something to add.

> Thanks for your help,
> 
> Jon

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: VHO-548969
Department: Support LDM
Priority: Normal
Status: Closed