[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [conduit] Slower transfer rates from NCEP

Carissa,


I based my 'no improvement' solely on the output of the traceroutes and mtr (which didn't really seem to change) and on the red line on that Unidata latency graph which had not changed.


I have since changed my CONDUIT requests to point to conduit.ncep.noaa.gov (first) and ncepldm4.woc.noaa.gov (second) and so far there seems to be a drastic improvement over what I was seeing last week. Looking at that same graph (time goes from left to right, so far left is about 3 days ago, right side is right now)


http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu


my latencies are now down around 60 seconds or so max, whereas last Friday they were over 3600s.  The red line on that graph appears to be related to my path through idd.unidata.ucar.edu, (since it's not there now that I'm not currently pulling CONDUIT from idd.unidata.ucar.edu) though for reasons I don't fully understand it didn't seem to affect my data delivery. The low latencies of the blue and green line below were making up for it and data was coming in completely and timely.


I don't understand why last Friday when this started, and I had requests to both conduit.ncep.noaa.gov and ncepldm4.woc.noaa.gov, I was seeing huge latencies, but when I changed to request from idd.unidata.ucar.edu (which had the red big lag) and ncepldm4.ncep.noaa.gov the data came in completely. It's almost as if when I had conduit.ncep.noaa.gov and ncepldm4.woc.noaa.gov in there last week, the ldm was not using/finding the lower latency site. Very strange and I don't understand it.


But bottom line is my transfer rates from conduit.ncep.noaa.gov DO seem to have greatly improved.


Thanks all for the investigation and work on this

Pete



--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden



From: Carissa Klemmer - NOAA Federal <address@hidden>
Sent: Thursday, February 25, 2016 11:50 AM
To: Pete Pokrandt
Cc: address@hidden; address@hidden; Gerry Creager - NOAA Affiliate
Subject: Re: [conduit] Slower transfer rates from NCEP
 
Pete,

Can you confirm that the slow downs to Boulder also started on Friday? And we'd appreciate testing conduit.ncep again. You are the first who has said they haven't seen an improvement. Before I report this I'd like to be 100% sure that both Boulder and conduit.ncep (Maryland) are bad.

Carissa Klemmer
NCEP Central Operations
Dataflow Team Lead
301-683-3835

On Thu, Feb 25, 2016 at 12:15 PM, Pete Pokrandt <address@hidden> wrote:

Here's an mtr output from about the last 16 hours or so, and also a traceroute from idd.aos.wisc.edu to conduit.ncep.noaa.gov. No real change as far as I can see..


mtr conduit.ncep.noaa.gov


 Host                                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu                                       0.0% 69823    3.3   0.6   0.5  67.2   5.2
 2. internet2-ord-600w-100G.net.wisc.edu                                                    0.0% 69823   18.0  18.0  18.0  92.5   5.3
 3. et-10-0-0.107.rtr.clev.net.internet2.edu                                                0.0% 69823   27.9  27.6  27.6 102.2   5.3
 4. et-11-3-0-1276.clpk-core.maxgigapop.net                                                 0.0% 69823   37.4  37.1  37.1 112.2   5.7
 5. noaa-i2.demarc.maxgigapop.net                                                           0.0% 69823   37.5  37.5  37.3 430.0  22.2
 6. 140.90.111.36                                                                           0.0% 69823   40.6  39.7  37.4 505.4  43.2
 7. 140.90.76.69                                                                            0.1% 69823   37.9  37.7  37.5 161.9   8.7
 8. ???


raceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte packets
 1  r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3)  1.115 ms  2.274 ms  1.083 ms
 2  internet2-ord-600w-100G.net.wisc.edu (144.92.254.229)  21.264 ms  21.261 ms  21.233 ms
 3  et-10-0-0.107.rtr.clev.net.internet2.edu (198.71.45.9)  32.370 ms  32.091 ms  31.255 ms
 4  et-11-3-0-1276.clpk-core.maxgigapop.net (206.196.177.4)  37.971 ms  38.678 ms  38.066 ms
 5  noaa-i2.demarc.maxgigapop.net (206.196.177.118)  37.997 ms  38.160 ms  38.298 ms
 6  140.90.111.36 (140.90.111.36)  41.625 ms  41.252 ms  41.509 ms
 7  140.90.76.69 (140.90.76.69)  41.837 ms  43.261 ms  42.738 ms
 8  * * *
 9  * * *
10  * * *

The latency graph for our conduit data hasn't changed either - we still see the ekg looking lags coming through ncep-ldm0.ncep.boulder_v_idd.uni


There are no times on that graph, but it covers roughly 3 days - each latency peak corresponds with a 6 hourly NCEP model run suite.

I have not tried actually getting CONDUIT data from conduit.ncep.noaa.gov since the slowness started last Friday. I'll try again pulling data from there and let you know what happens.

Pete




--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden