Re: does LDM lock files in transmission?

One trick would be make your pqact.conf entries pipe the 
bulletin to an executable script.  The script can take its 
STDIN and write it to a file, then touch another file or 
whatever you want to use as a signal.  Or if your 
application is small enough, pqact.conf can pipe the 
bulletin directly to your application.  This eliminates the 
need to configure pqact.conf to write the bulletin to the 
filesystem directly.

Andy Huhn
DataSwitch Information Services, Inc.
http://www.stormwarn.com

-----Original Message-----
[mailto:owner-ldm-users@xxxxxxxxxxxxxxxx]On 
Behalf Of Anne Wilson
Sent: Friday, October 05, 2001 5:34 PM
Cc: ldm-users@xxxxxxxxxxxxxxxx


"Jason J. Levit" wrote:

  Hi all,

  I'm writing an application that will process 
some real-time data I'm
ingesting through LDM.  This application will 
run around the clock in
the cron, so it needs to be fairly robust.  
Here's my question - say I
ingest a file via LDM.  How do I know that file 
is complete?  Does LDM
place some sort of lock or other information on 
the file before it's
completed writing?  If my application tries to 
access a file that does
not contain all the necessary data (a file that 
still has yet to be
completely transmitted, for example), then I 
need to have my application
not process the file.

  Does anyone else out there have experience 
with this?  I know there's
tricks you can perform, such as checking to see 
if the file has been
modified in the past 30 seconds before opening, 
etc., etc., but I was
just wondering if LDM actually places some sort 
of information with the
file that says "hey, this file isn't done yet". 
 Thanks for any help!

  Jason

--

--------------------------------------------------
--------------------------
Jason J. Levit, N9MLA                   
Research Scientist,
jlevit@xxxxxx             Center for Analysis 
and Prediction of Storms
Room 1022                              
University of Oklahoma
405/325-3503                           
http://www.caps.ou.edu/

Hi Jason,

I'm not sure if you're talking about when a 
product is received in the
queue, or when a product is being written out to 
disk from the queue via
pqact, although I'm guessing it's the latter.

When the LDM receives a product in the queue, 
that product is not
available to pqact or for retransmission to 
another site until the
product is complete.  For the purpose of 
retransmission or actions by
pqact, you can consider the product to be an atomic unit.

When pqact is writing the product out to file, it 
is unknown as to when
the product is complete.  (I am assuming that 
you're not talking about
appending to a file.)  This is a frequently asked 
question.  You would
have to devise some trick to figure that out.  
For example, if you know
the structure of the product, you could write a 
program that simply
writes out the product, but also identifies when 
the end of the product
is reached and touches some other file to serve 
as a flag.  You would
then PIPE the product to your program via pqact.  
But, to answer your
question, the LDM does not have any way to signal 
that pqact has or has
not completely written the product to disk.

Hope this helps!

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
anne@xxxxxxxxxxxxxxxx                  P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************

  • 2001 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: