[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #LVV-942838]: pqact.conf or ldmd.conf action or request to include product origin



Hi Jerry,

re:
> We have a user that gets MRMS 2D radar products from two sources.

Is the user using the LDM to get the MRMS products?  If yes, do you
know specifics on how the upstream sites are getting the products and
making them available via the LDM?

re:
> They want to know
> if there is a way to get the product source so that they can avoid a
> race condition.
> 
> From the user:
> 
> The only problem is that files sometimes have the exact same filename and
> arrive very close in time or at exactly the same time. This causes problems
> when our listener tries to do the perform the same operation on a file that
> is being operated on.

Assuming that the products are being received by the user's LDM (I assume so)
it should mean that the product MD5 signatures are different even when the
names of the files are the same.  If the MD5 signatures were the same, the
product received second would be discarded by the downstream LDM as one with
the same signature would already be in its LDM queue.

re:
> I noticed that there is a product origin (-O) option in pqcat, which gives
> the machines of origin with the product name and time (vm-bldr-mrms =
> Boulder; vm-lnx-mrms = College Park). Is there a way to obtain this
> information for use in the pqact.conf or ldmd.conf so that when we pipe the
> MRMS grib2s to a script, we can know which feed it came from?

Hmm... this one Steve will have to answer.  I assume by asking this you
are really trying to find a foolproof way of keeping the processing of
duplicately named products from stepping on one another.

re:
> Thanks for any help you can provide.

The reason I asked about exactly how the two upstreams are getting and making
the MRMS products available is as follows: if the upstreams are getting the
products by some means outside of the LDM and NOAAPort ingest, then they
could change the way that they are calculating the MD5 signature associated
with the MRMS products they are making available.  Since your/their comment
is that "files sometimes have the exact filename", having 'pqinsert' (or
'gribinsert') use the file name to calculate the MD5 signature would allow
the downstream LDM to eliminate the duplicate product before it was
added to its queue.  If, on the other hand, the products really are different
(e.g., as measured by their MD5 signatures), then it is up to the user to decide
which product is the one desired.  If the products are received far enough
apart time wise (meaning not in the same second), one could include the
system clock's second in the name of the file to be processed.

Another approach that comes to mind is to create a script to process the
products, and have the script write the product to a disk file in a
way that the name is unique.  The easiest way to do this is to include
the script's process ID as part of the disk file's name.

Again, Steve will have to chime in on whether or not the origin of the
product is available to a pattern-action file action.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: LVV-942838
Department: Support LDM
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.