[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Staging #FMV-140791]: Fwd: LDM data getting behind

Hi Stacey,

We drove your questions to Steve into our inquiry tracking system so
that more than one of us could take a look...

re:
> We have a computer that is processing local radar data from NOAAPort.
> 
> We notice from time to time that the data gets behind.  This seems to
> happen when there is a lot of weather occurring so there is more data
> being pushed through NOAAPort.  It is happening today since we have some
> bad weather in Kentucky.  What happens is that the data will slowly get
> further and further behind.  It got to 10 minutes behind this morning.

I am not entirely sure by what you mean when you say "the data will get
slowly get further and further behind".

Questions:

- do you mean that the product latency seen in the feed(s) slowly
  gets larger and larger?

- or, do you mean that the processing of data in the LDM queue gets
  further and further behind?

re:
> What we don't understand is what is causing this to occur.  The machine
> processing this data has hardly any load at all.  It is an extremely fast
> machine and during the times when the data gets behind, it is only using
> about 8-10% of it's CPU and the IO wait is at about 6-7%.

It certainly seems that the problem should not be with the machine that
is running your LDM.

re:
> If I do an
> "ldmamdin stop" and then "ldmadmin start", then the data gets caught
> backup and slowly starts getting behind again over the next couple hours.

This is very weird indeed.  The reason I say this is that the LDM will
request data from its upstream(s) from the point it last left off.
Stopping and restarting the LDM should not have any effect on
how fast products will be processed out of the queue (unless your
machine was totally bound up which does not seem to be the case given
your comments on how the machine is idling).

Question:

- where does the data come from?

  What I mean is: is/are your upstream(s) local, or are they somewhere
  else.

  The following may be a stretch:

  If the upstream feed host(s) are not in your domain, then a process
  doing "deep packet inspection" on all products being sent to you
  may get further and further behind.  This sort of artificial
  throttling is what we generally refer to as packet shaping.

re:
> Do you have any idea where the bottleneck would be that is causing the
> data to get behind like this?

Packet shaping?

re:
> I don't think it is the machine since it
> has almost no load at all.

I agree.  A machine that is idling should not experience this
sort of slowdown.

re:
> Are there any LDM enhancements/tweaks that
> you might know of that would speed things up?

We don't know enough about your situation to even guess what
kinds of tweaks might be useful.  We first need to know some
specifics of your setup:

- who are your upstream feed host(s)?

- how are you REQUESTing data from your upstream feed hosts(s)?

  Meaning, do you have one ~ldm/etc/ldmd.conf REQUEST line, or
  do you have several (and what are they)?

- how have you setup your local processing?

  The idea here is that if you have a single instance of 'pqact'
  processing everything from NOAAPort via a single pattern-action
  file, you may be causing a bottleneck.  I don't, however, see
  how an LDM stop/restart would alleviate that bottleneck since
  your LDM should get even more data when it restarts as it needs
  to work through the backlog of products that are waiting wile
  the LDM is down.

- what kind of machine is your LDM running on, and what are
  its specifics (CPU(s), RAM, OS, OS version, etc.)?

  This one might be crucial.  We have seen problems with the LDM
  running under MacOS-X.  The problem is not with the LDM code,
  but, rather, with the operating system.  Steve tried to get
  Apple interested enough to fix the problem (the problem did/does
  not exist in MacOS-X versions 10.4 or older), but was unable
  to capture anyone's attention for long enough to get the problem
  solved.

- are you reporting real time stats back to us?

  I.e., are you running the 'exec rtstats...' line in your
  ~ldm/etc/ldmd.conf file?

  If not, we will not be able to take advantage of the latency
  information that rtstats reports.

re:
> Your help is greatly appreciated.

No worries.  We will try to help out, but we need more information
before we can go any further.


Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: FMV-140791
Department: Support LDM
Priority: Normal
Status: Closed