[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #VTV-967500]: 'Duplicate' data from two LDM servers



Brice,

[I've driven your inquiry into our support email tracking system so that others 
might benefit (after removing your contact information, of course). I hope you 
don't mind.]

> I’m working on a ‘funny’ that is occurring in an LDM-data delivery
> prototype and in my research I came across this post
> http://www.unidata.ucar.edu/support/help/MailArchives/ldm/msg06314.html,
> which is similar to what I’m seeing, but ‘reversed’.  I’m requesting the
> same data from two servers.  I can see, using pqcat –vO, that the feeds are
> swapping from time to time.  Only I’m not losing data like Karen was, I’m
> getting ‘duplicate’ data.  The files have a time tag in their name and I
> can see them come in out of sequence and get filed.  So here is my theory
> and you can shoot holes in it.  Our LDM queue is 2.0G, but we are running
> all of the NOAAPort stream through it, plus a couple of other continuous
> data streams.  My theory is that the upstream server(s) don’t have as much
> traffic on them and when our client swaps LDM gets older data files from
> the ‘other’ server, but doesn’t flag them as duplicates, because they have
> already left our queue.  I have run part of a test on this and, if I only
> access one server, I don’t see the problem.

Sounds very plausible.

> If my theory sounds right to you, then how would I go about fixing this,
> would you know?  We have not had Primary/Secondary servers before, so I’m
> not sure about how to go about setting that up or if it would solve the
> problem.  We are *providing* now prime and secondary servers to some of our
> customers, so this info would be useful in that direction as well.

The main concept is to have the product-queue be large enough so that the least 
amount of time that any product spends in the queue is greater than the 
"maximum latency" of the REQUEST-s (which is one hour by default). This will 
ensure that the product-queue can reject any duplicates. You can see the age of 
the currently oldest product in the queue via the command "pqmon" (it's the 
last field, in seconds).

Beginning with LDM 6.9.0, the LDM system can automatically reconcile the size 
of the product-queue and the maximum latency parameter to ensure duplicate 
rejection. Before this version, such reconciliation required manual 
intervention. What version do you have?

If your version is 6.9.0 or later, then set the registry parameter 
"/reconciliation-mode" to either "increase queue" or "decrease maximum latency" 
and then periodically execute the command "ldmadmin check" or "ldmadmin 
vetqueuesize".

More information on this can be found at 
<http://www.unidata.ucar.edu/software/ldm/ldm-current/basics/LDM-registry.html> 
and on the ldmadmin(1) manual-page ("man ldmadmin").

> Thanks for the assist as always,

Always our pleasure.

> Brice
> 
> Brice Biggerstaff, CISSP
> Johnson Space Center
> Weather Decision Support System
> MIDDS Software Support Lead
> XXX-XXX-XXXX (w)
> XXX-XXX-XXXX (p)
> address@hidden  (alpha pager for text and email)
> 
> *Res Confacti Erimus*
> *“We Get Things Done!”*

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: VTV-967500
Department: Support LDM
Priority: Normal
Status: Closed