[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #QCK-282686]: Implementation of redundant connections

Hi Joe,

re:
> Thanks for getting back to me.

No worries.

re:
> The problem that I'm trying to get around is this:
> 
> Our data provider has two datacenters, each producing a particular
> radar file for us.
> 
> Each datacenter has an LDM that we get data from.  Each datacenter
> produces a file, however, due to variations in processing, the timestamp
> of the file could be a minute or two off:
> 
> Datacenter1 --> radar.09212018.151400.nc -->  LDM1
> Datacenter2 --> radar.09212018.151500.nc -->  LDM2

Hmm... this is a slightly different situation that I had in mind when I
wrote my original response.  The "ping ponging" of feeds working
depends on the feed REQUESTs being identical AND the contents of the
feeds being identical wrt to the products MD6 signatures.  If the
products in one feed are different from the products in the other feed
(meaning the MD5 signatures are different), then the ping ponging effect
is _NOT_ what you want at all since the promotion of a secondary
feed to primary is a function of how many products from the
secondary feed are inserted into the local LDM queue over the sampling
period, and that is a function of the MD5 signature of the products
received.  If a product with a particular MD5 signature is received in
the primary feed, and it successfully gets inserted into the local LDM
queue (because no product already in the queue has the same MD5 signature),
and then a product with the same MD5 signature is received in the secondary
feed, the second product will be rejected by the LDM queue code since
a product with the same MD5 signature already exists in the local LDM queue.

In the case you are describing, I would force data from both upstreams
to be sent in primary mode and then let the LDM product queue code do
duplicate product detection and rejection:

Request EXP ".*"  IPAddress1
Request EXP "(.*)"  IPAddress2

The addition of the parentheses in the second REQUEST makes the extended
regular expressions different, so each connection will operate in
primary mode.

re:
> This is the reason why I only wanted to receive files from only 1 LDM
> at a time. (we have pqact set to throw out dupes, but in this case, it won’t).

Hmm...

re:
> I think the only real solution is for the provider to give us files
> that aren’t timestamped, which will enable the LDM to keep things in order.

The LDM will "keep things in order" when the MD5 signatures for the products
are calculated equivalently.  You (the originator) can either use the entire
content of the product to calculate the MD5 signature, or you (the originator)
can just use the Product ID to calculate the MD5 signature.  Since we have only
been talking about file names, I don't know if the LDM/IDD Product IDs are
different or not.

re: 
> We’re running LDM 6.13.6.  I don’t think there’s anything to do, but
> you guys know far more than me about LDM and implementations, so I was
> throwing a hailmary your way.

:-)  Have my comments helped?

re:
> And yes, I have no desire to switch between conf files.

OK.

re:
> That’s another variable that I do not want to deal with.

I understand completely.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: QCK-282686
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.