Re: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu starting the last day or two.

  • To: Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>, Arthur A Person <aap1@xxxxxxx>, "_NCEP.List.pmb-dataflow" <ncep.list.pmb-dataflow@xxxxxxxx>
  • Subject: Re: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu starting the last day or two.
  • From: Pete Pokrandt <poker@xxxxxxxxxxxx>
  • Date: Fri, 19 Feb 2016 19:52:10 +0000
  • Authentication-results: albany.edu; dkim=none (message not signed) header.d=none; albany.edu; dmarc=none action=none header.from=aos.wisc.edu;
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:23
An update - just before noon CST I updated to the newest ldm (6.12.14) and also 
switched my CONDUIT request sites so I'm requesting from idd.unidata.ucar.edu 
and ncepldm4.woc.noaa.gov (both in Boulder, I believe) and commented out the 
request to conduit.ncep.noaa.gov (in Washington DC?)


Latencies are almost immediately better (see attached gif) and the 18 UTC NAM 
seems to be coming in fine. Guess I'm good for now, though me feeding CONDUIT 
from idd.unidata.ucar.edu doesn't seem ideal.. Would be better to figure out 
why I'm getting such latencies from the two noaa sites.




Some mtr data:


idd.aos.wisc.edu to idd.unidata.ucar.edu:


                                                        My traceroute  [v0.75]
idd.aos.wisc.edu (0.0.0.0)                                                      
                             Fri Feb 19 13:48:54 2016
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                
             Packets               Pings
 Host                                                                           
           Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu                           
            0.0%  6291    6.9   0.9   0.5  49.6   3.7
 2. internet2-ord-600w-100G.net.wisc.edu                                        
            0.0%  6291   18.2  18.1  18.0  70.9   3.8
 3. ae0.3454.core-l3.frgp.net                                                   
            0.0%  6291   40.2  40.2  39.9  91.3   4.0
 4. tcom-gs-1-vl873.ucar.edu                                                    
            0.0%  6291   41.4  41.1  40.8 160.8   6.6
 5. flra.ucar.edu                                                               
            0.0%  6291   41.5  45.7  40.9 406.8  22.9
 6. idd.unidata.ucar.edu                                                        
            0.2%  6290   41.8  41.1  40.9 106.1   4.8





idd.aos.wisc.edu to ncepldm4.woc.noaa.gov


                                       My traceroute  [v0.75]
idd.aos.wisc.edu (0.0.0.0)                                                 Fri 
Feb 19 13:49:42 2016
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                           Packets              
 Pings
 Host                                                    Loss%   Snt   Last   
Avg  Best  Wrst StDev
 1. r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu     0.0%  7014    1.3   
0.9   0.6  50.0   3.8
 2. internet2-ord-600w-100G.net.wisc.edu                  0.0%  7014   18.7  
18.2  18.1  67.9   3.7
 3. ae0.3454.core-l3.frgp.net                             0.0%  7014   54.7  
40.3  40.0  90.9   4.0
 4. noaa-i2.frgp.net                                      0.0%  7014   50.0  
41.4  40.8 106.0   5.7
 5. 2001-mlx8-eth-1-2.boulder.noaa.gov                    0.0%  7014   58.3  
42.2  40.8 107.8   5.6
 6. mdf-rtr-6.boulder.noaa.gov                            0.0%  7014   56.9  
42.3  40.9 107.0   5.7
 7. ???





idd.aos.wisc.edu to conduit.ncep.noaa.gov


                                     My traceroute  [v0.75]
idd.aos.wisc.edu (0.0.0.0)                                                 Fri 
Feb 19 13:49:18 2016
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                           Packets              
 Pings
 Host                                                    Loss%   Snt   Last   
Avg  Best  Wrst StDev
 1. r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu     0.0%  7031    1.6   
0.8   0.6  50.6   3.6
 2. internet2-ord-600w-100G.net.wisc.edu                  0.0%  7031   18.2  
18.1  18.0  68.7   3.6
 3. et-10-0-0.107.rtr.clev.net.internet2.edu              0.0%  7031   27.9  
27.8  27.6  76.4   3.8
 4. et-11-3-0-1276.clpk-core.maxgigapop.net               0.0%  7031   37.3  
37.3  37.1  86.1   4.3
 5. noaa-i2.demarc.maxgigapop.net                         0.0%  7031   39.0  
40.6  37.3 373.8  20.6
 6. 140.90.111.36                                         0.0%  7031   37.6  
53.5  37.4 468.2  34.2
 7. 140.90.76.69                                          0.0%  7031   37.8  
38.6  37.5 130.1   6.6
 8. ???






Pete



<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - poker@xxxxxxxxxxxx



________________________________
From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> on 
behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
Sent: Friday, February 19, 2016 11:20 AM
To: Carissa Klemmer - NOAA Federal; Arthur A Person; _NCEP.List.pmb-dataflow
Cc: Bentley, Alicia M; Daes Support; support-conduit@xxxxxxxxxxxxxxxx; Michael 
Schmidt
Subject: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu 
starting the last day or two.


All,


Not sure if this is on my end or somewhere upstream, but the last several runs 
my CONDUIT latencies have been getting huge to the point where we are losing 
data.


I did stop my ldm the other day to add in an alternate feed for Gilbert at 
allisonhous.com, not sure if that pushed me over a bandwidth limit, or by 
reconnecting we got hooked up to a different remote ldm, or taking a different 
path, that shot the latencies up.


Seems to be really only CONDUIT, none of our other feeds show this kind of 
latency.


Still looking into things locally, but wanted make people aware. I just 
rebooted idd.aos.wisc.edu, will see if that helps at all.


Here's an ldmping and traceroute from idd.aos.wisc.edu to conduit.ncep.noaa.gov.


[ldm@idd ~]$ ldmping conduit.ncep.noaa.gov
Feb 19 17:16:08 INFO:      State    Elapsed Port   Remote_Host           
rpc_stat
Feb 19 17:16:08 INFO: Resolving conduit.ncep.noaa.gov to 140.90.101.42 took 
0.00486 seconds
Feb 19 17:16:08 INFO: RESPONDING   0.115499  388   conduit.ncep.noaa.gov



traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte 
packets
 1  r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3)  0.760 ms  
0.954 ms  0.991 ms
 2  internet2-ord-600w-100G.net.wisc.edu (144.92.254.229)  18.119 ms  18.123 ms 
 18.107 ms
 3  et-10-0-0.107.rtr.clev.net.internet2.edu (198.71.45.9)  27.836 ms  27.852 
ms  27.838 ms
 4  et-11-3-0-1276.clpk-core.maxgigapop.net (206.196.177.4)  37.363 ms  37.363 
ms  37.345 ms
 5  noaa-i2.demarc.maxgigapop.net (206.196.177.118)  38.051 ms  38.254 ms  
38.401 ms
 6  140.90.111.36 (140.90.111.36)  118.042 ms  118.412 ms  118.529 ms
 7  140.90.76.69 (140.90.76.69)  41.764 ms  40.343 ms  40.500 ms
 8  * * *
 9  * * *
10  * * *

Similarly to ncepldm


[ldm@idd ~]$ ldmping ncepldm4.woc.noaa.gov
Feb 19 17:18:40 INFO:      State    Elapsed Port   Remote_Host           
rpc_stat
Feb 19 17:18:40 INFO: Resolving ncepldm4.woc.noaa.gov to 140.172.17.205 took 
0.001599 seconds
Feb 19 17:18:40 INFO: RESPONDING   0.088901  388   ncepldm4.woc.noaa.gov
^C

[ldm@idd ~]$ traceroute ncepldm4.woc.noaa.gov
traceroute to ncepldm4.woc.noaa.gov (140.172.17.205), 30 hops max, 60 byte 
packets
 1  r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3)  0.730 ms  
0.831 ms  0.876 ms
 2  internet2-ord-600w-100G.net.wisc.edu (144.92.254.229)  18.092 ms  18.092 ms 
 18.080 ms
 3  ae0.3454.core-l3.frgp.net (192.43.217.223)  40.196 ms  40.226 ms  40.256 ms
 4  noaa-i2.frgp.net (128.117.243.11)  40.970 ms  41.012 ms  40.996 ms
 5  2001-mlx8-eth-1-2.boulder.noaa.gov (140.172.2.18)  42.780 ms  42.778 ms  
42.764 ms
 6  mdf-rtr-6.boulder.noaa.gov (140.172.6.251)  40.869 ms  40.922 ms  40.946 ms
 7  * * *
 8  * * *




Pete




<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - poker@xxxxxxxxxxxx

Attachment: Screenshot at 2016-02-19 13:46:11.png
Description: Screenshot at 2016-02-19 13:46:11.png