[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #ZAJ-969398]: IDD Data is too old

Hi,

> The hostname in the registry is "irads-ingest0.net.ou.edu".

That's the name that appears on the rtstats(1) webpage, 
<http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex>, which is the 
*only* thing that name is used for, which I why we need the IP address.

> In the registry, we have 'time-offset' lowered to 600 seconds. Even with that 
> low of a
> period to backfill, it never catches up. In fact, it gets further behind.

If, on average, products want to come in faster than the LDM can insert them 
into the queue, then latency will increase regardless of the size of the queue. 
Something is slowing how fast your LDM can insert data-products. Possibilities 
include 1) a really slow system trying to do too much (not likely, IMHO); and 
2) some network problem that's reducing the effective bandwidth of the TCP 
connections (far more likely, IMHO).

> Yesterday evening, I tested idd.meteo.psu.edu. The same issue occurred. We 
> were able to
> get a bit better performance by splitting the request into two, but it still 
> doesn't
> perform well.

That's consistent with both hypotheses above.

> While we've been focused on one host, irads-ingest0, at 156.110.246.56, we 
> are seeing the
> same issue from echo-ingestA.services.ou.edu (129.15.2.32). It has been 
> tested against
> idd.unidata.ucar.edu and idd.aos.wisc.edu and behaves the same as 
> irads-ingest0. To add
> to the issue, both hosts have started having issues receiving the NL2 TDS 
> feed as well.

That's consistent with the network hypothesis but not with the slow system 
hypothesis.

> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD3+echo-ingesta.services.ou.edu
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD2+echo-ingesta.services.ou.edu
> 
> The two hosts are in spate networks

spate?

> on separate campuses, managed by different parts of
> the IT department. With that, we have put in a ticket with our ISP, OneNet. 
> They have
> requested traceroutes from our upstream sources to our hosts, if possible, to 
> trace the
> data path.
> 
> Would it be possible to get traceroutes from a host behind 
> idd.unidata.ucar.edu to
> 129.15.2.32 and 156.110.246.56?

Here you go:

~: traceroute 129.15.2.32
traceroute to 129.15.2.32 (129.15.2.32), 30 hops max, 60 byte packets
 1  flr-n140.unidata.ucar.edu (128.117.140.251)  0.660 ms  1.127 ms  1.572 ms
 2  ml2core-fl2core.unet.ucar.edu (128.117.243.194)  1.101 ms  1.104 ms  1.496 
ms
 3  corel3-ml2core-i2.unet.ucar.edu (128.117.243.141)  1.519 ms  1.553 ms  
1.541 ms
 4  v3454.rtr-chic.frgp.net (192.43.217.222)  23.119 ms  23.145 ms  23.130 ms
 5  et-2-1-0.4079.rtsw.chic.net.internet2.edu (162.252.70.116)  23.693 ms  
23.691 ms  23.806 ms
 6  ae-3.4079.rtsw.kans.net.internet2.edu (162.252.70.141)  34.436 ms  34.440 
ms  34.475 ms
 7  et-7-0-0.4079.rtsw.tuls.net.internet2.edu (162.252.70.35)  38.431 ms  
38.370 ms  38.334 ms
 8  198.71.46.45 (198.71.46.45)  38.658 ms  38.659 ms  38.642 ms
 9  164.58.244.44 (164.58.244.44)  38.614 ms  38.718 ms  38.448 ms
10  164.58.244.15 (164.58.244.15)  41.176 ms  41.325 ms  41.317 ms
11  164.58.245.55 (164.58.245.55)  40.658 ms  40.803 ms 164.58.245.53 
(164.58.245.53)  40.984 ms
12  164.58.245.56 (164.58.245.56)  41.402 ms 164.58.245.58 (164.58.245.58)  
40.972 ms  40.969 ms
13  164.58.244.33 (164.58.244.33)  41.475 ms  41.507 ms  41.496 ms
14  164.58.10.98 (164.58.10.98)  41.416 ms  41.593 ms  41.586 ms
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
~: traceroute 156.110.246.56
traceroute to 156.110.246.56 (156.110.246.56), 30 hops max, 60 byte packets
 1  flr-n140.unidata.ucar.edu (128.117.140.251)  0.714 ms  0.873 ms  0.865 ms
 2  ml1core-flacore.unet.ucar.edu (128.117.243.78)  0.949 ms  0.951 ms  1.346 ms
 3  ml2core-ml1core.unet.ucar.edu (128.117.243.99)  0.915 ms 
ml2core-fl2core.unet.ucar.edu (128.117.243.194)  1.368 ms  1.363 ms
 4  corel3-ml2core-i2.unet.ucar.edu (128.117.243.141)  1.710 ms  1.710 ms  
1.690 ms
 5  v3454.rtr-chic.frgp.net (192.43.217.222)  23.139 ms  23.077 ms  23.075 ms
 6  et-2-1-0.4079.rtsw.chic.net.internet2.edu (162.252.70.116)  23.555 ms  
23.538 ms  23.594 ms
 7  ae-3.4079.rtsw.kans.net.internet2.edu (162.252.70.141)  34.683 ms  34.676 
ms  34.463 ms
 8  et-7-0-0.4079.rtsw.tuls.net.internet2.edu (162.252.70.35)  38.283 ms  
40.679 ms  40.648 ms
 9  198.71.46.45 (198.71.46.45)  39.418 ms  39.410 ms  39.389 ms
10  164.58.244.44 (164.58.244.44)  39.205 ms 164.58.244.46 (164.58.244.46)  
38.648 ms 164.58.244.44 (164.58.244.44)  38.710 ms
11  164.58.244.217 (164.58.244.217)  38.561 ms  38.554 ms  38.532 ms
12  164.58.16.26 (164.58.16.26)  38.372 ms  38.370 ms  38.885 ms
13  156.110.254.97 (156.110.254.97)  46.317 ms  45.003 ms  45.072 ms
14  156.110.254.50 (156.110.254.50)  41.374 ms 156.110.254.62 (156.110.254.62)  
44.831 ms 156.110.254.50 (156.110.254.50)  41.488 ms
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
~: 

It should also be possible to run traceroute on those systems to 
idd.unidata.ucar.edu.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: ZAJ-969398
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.