[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #QMV-683924]: Delay in LDM data

Hi Justin,

re:
> I recently noticed an issue regarding the relay of LDM data sets. We
> have a remote LDM machine that sends data out to other off-site
> LDM machines. We noticed a delay in data on the receiving LDM
> machine. Eventually after about 10 minutes, the data starts flowing
> over ldmadmin watch. Then several minutes later, the data actually
> starts writing to disk on the receiving machines. Small data sets such
> as IDS/DDPLUS are the first data sets to be received (after 5 minutes)
> while larger data sets such as NOTHER are the last to come in... 10
> minutes later.

This sounds like there is either some deep packet inspection being
made for the traffic in the LDM connections between the upstream
and downstreams, or the available bandwidth is being throttled.  The
classic symptom of what we generically refer to as "packet shaping"
is "low" volume feeds showing small receipt latencies while "high"
volume feeds show significant latencies.

re:
> When we stop the LDM on the receiving machines and restart it, the data
> starts coming in instantaneously (maybe from the queue?).

When you say "starts coming in" are you saying that you start
seeing the products being inserted into the local, downstream LDM queue
(via an 'ldmadmin watch' or a 'notifyme -vl-', or are you saying that
you start seeing the products being processed to disk?

If you are saying the former, it sounds like there may be adaptive
"packet shaping" (artificial bandwidth limiting) that increases
as time goes on.  When the LDM is started/restarted, existing
connections will be broken and new ones created, so the data "coming
through" fast after a restart and then slowing down as time goes on
sounds like something is artificially slowing down the connection, and
deep packet inspection or something else is suggested.

re:
> Any ideas what would cause the initial delay in data receipt?

No, not really.  Can you setup a situation where you are monitoring the
LDM log files on the upstream and downstream LDMs simultaneously?

Another thought: are the system clocks on the upstream and downstream
LDMs synchronized?   If the time on the downstream LDM (the LDM that
REQUESTs data from the upstream) is ahead of the clock on the upstream,
then the REQUEST may be asking for data from a time that has not yet
occurred on the upstream.  We saw situations like this so frequently
in the past that the LDM was modified to check to see if the system
clock is accurate, and if it is not, to let the user know that corrective
action must be taken.

Questions:

- what version(s) of the LDM are you running on both the upstream and
  downstream LDMs?

- are the clocks on by upstream and downstream LDM(s) synchronized?

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: QMV-683924
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.