[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #ZAJ-969398]: IDD Data is too old

Hi Elliott,

re:
> Since yesterday afternoon, we have started seeing data from
> idd.unidata.ucar.edu being rejected because it is too old. I have checked
> the NEXRAD3 and NGRID feeds, but it is likely happening in others.

OK.

re:
> We are in Oklahoma on CST, so 14:19 CST is 20:19 UTC. The data below appears 
> to be just over an hour old:
> Nov  5 14:14:23 irads-ingest0 idd.unidata.ucar.edu[19432] INFO: Ignoring 
> too-old product:        279 20181105190549.724 NEXRAD3 82757661  SDUS54 KFWD 
> 051859 /pNVLDFW
> Nov  5 14:14:23 irads-ingest0 idd.unidata.ucar.edu[19432] INFO: Ignoring 
> too-old product:       6145 20181105190549.725 NEXRAD3 82757662  SDUS33 KBIS 
> 051859 /pN3SBIS
> Nov  5 14:14:23 irads-ingest0 idd.unidata.ucar.edu[19432] INFO: Ignoring 
> too-old product:       8300 20181105190549.741 NEXRAD3 82757663  SDUS76 KSGX 
> 051904 /pN0ZNKX
> Nov  5 14:14:23 irads-ingest0 idd.unidata.ucar.edu[19432] INFO: Ignoring 
> too-old product:       4107 20181105190549.742 NEXRAD3 82757664  SDUS54 KFWD 
> 051859 /pNCRDFW
> 
> Nov  5 14:19:05 irads-ingest0 idd.unidata.ucar.edu[19442] INFO: Ignoring 
> too-old product:       3016 20181105190006.857   NGRID 37304044  FOUS15 KWNO
> 051800 /pNBHUSA
> Nov  5 14:19:05 irads-ingest0 idd.unidata.ucar.edu[19442] INFO: Ignoring 
> too-old product:       3426 20181105190006.857   NGRID 37304045  FOUS15 KWNO
> 051800 /pNBHUSA
> 
> We saw latency drift upward yesterday afternoon till LDM started rejecting:
> 
> https://status.irads.ou.edu/images/L3latency.png
> 
> The green line is from the host connected to the IDD. The blue line is
> from a different host that receives data from a local NOAAPort that is
> functioning properly.

Our real-time stats pages also show that your feed latencies have grown:

Unidata HomePage
http://www.unidata.ucar.edu

  Data -> IDD Operational Status
  http://rtstats.unidata.ucar.edu/rtstats/

    Real-time IDD Statistics -> Statistics by Host
    http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex

      irads-ingest0.net.ou.edu [6.12.14]
      
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?irads-ingest0.net.ou.edu

The higher volume feeds from idd.unidata.ucar.edu are all showing unacceptably
high latencies.  For instance:

NEXRAD3
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD3+irads-ingest0.net.ou.edu

NGRID
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NGRID+irads-ingest0.net.ou.edu

CONDUIT
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+irads-ingest0.net.ou.edu

re:
> We aren't seeing any issues from other sources. We receive the NEXRAD2
> feed directly from TDS/RDS and those feeds are functioning properly
> as well.
> 
> Please let us know if this is a known issue or if there is something we
> need to do on our end.

We are not seeing anything out of the ordinary on the real-server backend
node of our top level IDD relay cluster, idd.unidata.ucar.edu, that is
feeding delta-ingest0.irads.ou.edu.  In fact, we are feeding NEXRAD3
and some other feeds on a single REQUEST to a machine at UIowa, and
its latencies are on the order of a second or two.

We noticed that the latencies for your 5-way split of CONDUIT feed
started getting bad (going south) yesterday, November 4 at approx.
9 UTC:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+irads-ingest0.net.ou.edu

The interesting thing is that the latencies dropped down to near zero for 
awhile,
increased again, then dropped between 18 and 21 UTC (approx) and then have
gotten steadily worse since.

Question:

- was anything done with the networking yesterday morning?

Suggested test:

- try changing your NEXRAD3 feed REQUEST from idd.unidata.ucar.edu
  to idd.meteo.psu.edu

  And, please let us know the results of this test.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ZAJ-969398
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.