[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #AYS-505040]: Recent NEXRAD2 IDD latencies with idd.unidata

Hi,

re:
> My local monitoring has been showing sporadic NEXRAD2 IDD latencies
> on the order of 600 seconds since Friday (9 Sept).  You can see these here:
> 
> https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD2+node0.unidata.ucar.edu
> 
> Are known troubles afoot?  

We have not seen any notifications related to the NEXRAD2 feed from NOAA,
and the flows out of our IDD relay clusters appears to be ioeratibg normall.

re:
I was feeding from idd.unidata, but have switched to iddb.unidata as it does not
> appear to have these issues.

Given the NEXRAD2 latency plots for randomly chosen real-servers backends of
the Iddb.unidata.ucar.edu cluster, which is housed in the main machine room at
the NCAR Mesa Lab, I would have said that the high latencies your monitoring
is showing would be worse, not better than nodes in the idd.unidata.ucar.edu
cluster, which is housed in the NCAR Wyoming Supercomputer Center in Cheyenne,
WY.  It could be that the products from stations whose high latencies were/are
triggering your alarms are not being inserted in the LDM queues on the IDDB
cluster backends, so your monitoring would not show them as being abnormally
high since they are never received.

I just checked the indicated NEXRAD2 latencies on the three accumulators
(the machines that make the REQUEST to upstreams) of our clusters, and
one is showing very high latencies for the NEXRAD2 feed while the other
two are not.  Exactly why this is happening is not known at the moment,
but we will investigate.  

Given the redundant feed REQUESTs that all of the backends of our clusters
have, it is most likely that the high latencies being seen are coming from
this one accumulator (oliver.unidata.ucar.edu), and those products may well
be "second trip" products, which is how we refer to products that are received
more than once but late enough that they are not ignored by the LDM duplicate
product detection and rejection mechanism.

Again, further investigations will be made...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: AYS-505040
Department: Support IDD
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.