[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #AIZ-345619]: LDM: pattern in ldmd.conf and pact.conf no longer works for CONDUIT??

Hi Christian,

vkepler mystery solved!

Here goes:

1) your LDM registry had the reconciliation mode set to 'decrease max latency'

   The result of this is the LDM automatically tuned the 'max-latency' and
   'time-offset' parameters and saved them in the registry.  The 'max-latency'
   value had been set down to 138 seconds, and this was causing the rejection
   (meaning not being inserted into the LDM queue) of all products whose latency
   exceeded 138 seconds.

   At the same time, the latency on CONDUIT products we are receiving from
   NCEP is hovering at values greater than about 150 seconds due to a clock
   problem on the NCEP top level source for CONDUIT -- NCEP has been informed
   about this problem more than once, and they have said that they will look
   into the problem, but so far nothing has been done.

2) the LDM queue size on vkepler is not large enough to be able to reject
   duplicate products when all of the feeds/products being REQUESTed are
   received

   We checked to see how much RAM vkepler has, and, unfortunately, it only
   seems to have 8GB, and there is also some swap being used, so increasing
   the size of the LDM queue is not really possible.

What we changed:

First, we changed the LDM registry reconciliation mode from 'decrease maximum 
latency'
to 'do nothing'.  Next, we changed the 'max-latency' and 'time-offset' values in
the registry to 3600 seconds just to see if CONDUIT products would be inserted 
into
the LDM queue, and they were.  After looking at the LDM metrics plots:

ldmadmin plotmetrics -b 20200706

and especially the age of the oldest product in the queue, we changed the 
'max-latency'
and 'time-offset' registry values to 2100 seconds.

At the same time, I commented out the CONDUIT REQUEST to autan and re-instituted
the REQUEST to iddb.unidata.ucar.edu.  After restarting the LDM we watched
products being inserted into the queue using 'ldmadmin watch', and we saw
CONDUIT products being put into the queue.

What needs to be done next:

The age of the oldest product in the LDM queue needs to be monitored to make
sure that it doesn't get too small.  If it does, the 'max-latency' and
'time-offset' values will need to be decreased, BUT they will need to be
kept larger than the typical CONDUIT latency values in order to actually
have CONDUIT products written to your LDM queue.  You can monitor the
CONDUIT latency by viewing a latency plot for any of our top level relay
cluster backend machines:

cluster0, ... cluster6 .unidata.ucar.edu
node0 .. node8         .unidata.ucar.edu

Recommendations:

- if it is possible to add more memory to vkepler, we recommend doing
  so AND increasing the LDM queue size

- if the reconciliation mode on autan.sca.uqam.ca is also set to
  'decrease maximum latency', we recommend changing it to 'do nothing'
  and reviewing the 'max-latency' and 'time-offset' values to make
  sure that they are large enough to receive CONDUIT products

Final comment:

We will be contacting NCEP to once again point out that they have a problem,
and that problem is affecting sites receiving the data.

Please let us know if you have any questions/comments...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: AIZ-345619
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.