"Jason J. Levit" wrote:
>
> Hi all,
>
> I'm writing an application that will process some real-time data I'm
> ingesting through LDM. This application will run around the clock in
> the cron, so it needs to be fairly robust. Here's my question - say I
> ingest a file via LDM. How do I know that file is complete? Does LDM
> place some sort of lock or other information on the file before it's
> completed writing? If my application tries to access a file that does
> not contain all the necessary data (a file that still has yet to be
> completely transmitted, for example), then I need to have my application
> not process the file.
>
> Does anyone else out there have experience with this? I know there's
> tricks you can perform, such as checking to see if the file has been
> modified in the past 30 seconds before opening, etc., etc., but I was
> just wondering if LDM actually places some sort of information with the
> file that says "hey, this file isn't done yet". Thanks for any help!
>
> Jason
>
> --
> ----------------------------------------------------------------------------
> Jason J. Levit, N9MLA Research Scientist,
> jlevit@xxxxxx Center for Analysis and Prediction of Storms
> Room 1022 University of Oklahoma
> 405/325-3503 http://www.caps.ou.edu/
Hi Jason,
I'm not sure if you're talking about when a product is received in the
queue, or when a product is being written out to disk from the queue via
pqact, although I'm guessing it's the latter.
When the LDM receives a product in the queue, that product is not
available to pqact or for retransmission to another site until the
product is complete. For the purpose of retransmission or actions by
pqact, you can consider the product to be an atomic unit.
When pqact is writing the product out to file, it is unknown as to when
the product is complete. (I am assuming that you're not talking about
appending to a file.) This is a frequently asked question. You would
have to devise some trick to figure that out. For example, if you know
the structure of the product, you could write a program that simply
writes out the product, but also identifies when the end of the product
is reached and touches some other file to serve as a flag. You would
then PIPE the product to your program via pqact. But, to answer your
question, the LDM does not have any way to signal that pqact has or has
not completely written the product to disk.
Hope this helps!
Anne
--
***************************************************
Anne Wilson UCAR Unidata Program
anne@xxxxxxxxxxxxxxxx P.O. Box 3000
Boulder, CO 80307
----------------------------------------------------
Unidata WWW server http://www.unidata.ucar.edu/
****************************************************
