[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #BMK-187963]: LDM Redundancy and Duplicated Files

Hi,

re:
> I'm a part of the Weather team at the Johnson Space Center in
> Houston.  Our weather system uses multiple LDMs to obtain data
> from external sites and redundancy.   We have an issue with LDM
> redundancy and duplicated files.  Please see the below diagram
> for our current LDM layout and the data flow (from left to right).\
> The question is as stated in the diagram below:

To answer the question posed in the diagram you sent, an LDM will
reject duplicate products if the duplicates are received while the
original one(s) is(are) still in the LDM queue.

NB: the LDM rejects duplicate products by comparing the MD5 signatures
that are associated with each product.  If the MD5 signatures differs
at all, the LDM will see them as being the same.

Given how the LDM reject duplicates, the question is how the MD5 signatures
are being  calculated for products originating from LDM1 and LDM2?
The default way that an MD5 signature is calculated for a product (and
that occurs when the product is first inserted into an LDM queue) is
to use all of the bytes in the product in the calculation.  If one
is using the LDM utility 'pqinsert' to insert the products into the
originating LDM queue, the calculation of the MD5 signature can either
be set as the default or only using the characters in the product's
Product ID.  Since your LDMA and LDMB machines are seeing the "same"
products as being different, then it is most likely that they are
either a) not identical or b) have their MD5 signatures calculated
differently.

re:
> Please let me know what you think.  This file duplication issue is
> a major issue for our weather data processing so we appreciate you inputs.

Again, the question (posed above) is how the MD5 signatures are being
calculated when the products are being inserted into the first LDM
queue?

The second question is how long after is a duplicate product received
by either/both LDMA or LDMB?  Associated with that question is what
is the residency time of products in LDMA's and LDMB's queues?

If you turned on LDM metrics gathering (via a cron entry), you will
have a record of the age of the oldest product in the local LDM
queue as a function of time.  The entries most sites setup for doing
this monitoring looks like:

#
# LDM-6.8.1+ metrics gathering
#
* * * * * bin/ldmadmin addmetrics > /dev/null
0 0 1 * * bin/ldmadmin newmetrics > /dev/null

The last question for this email is have you setup metrics gathering
on either/both LDMA or/and LDMB?

re:
> Thank you in advance!

No worries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: BMK-187963
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.