Re: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu starting the last day or two.

  • To: Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>
  • Subject: Re: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu starting the last day or two.
  • From: Karen Cooper - NOAA Affiliate <karen.cooper@xxxxxxxx>
  • Date: Mon, 22 Feb 2016 09:06:45 -0600
NSSL uses the Boulder TIC, so it may have been a Colorado problem.   Also,
our problems actually started on Thursday -- (my memory is getting bad).

I switched us over to a back up IRADS server for data, and the problem has
not recurred.

The problem is, that the latencies were intermittent.  I at first thought
it was just a few hours early Thursday morning (11Z-13Z), but then it
happened again mid-day Thursday (17Z-19Z).   Which makes me wonder if it's
model data or something saturating the network and throttling us.

I then made a switch to the OU IRADS backup server, and am not sure if I
would have seen it again.

This morning I started up one of my backup servers, and now I'm connected
to both of OU's IRADS servers separately to see if either or both of them
are still having the issue.  I plan to leave it running that way for most
of the day today for testing.

On Mon, Feb 22, 2016 at 8:49 AM, Carissa Klemmer - NOAA Federal <
carissa.l.klemmer@xxxxxxxx> wrote:

> Karen,
>
> There was definitely something going on outside of NCEPs servers. We heard
> from a couple of folks from Colorado to Oregon having issues that moved
> their services to a new location and rates went back to normal. Since you
> are in Ok it may have been further reaching into the mid-west than we
> heard. Are you still having issues? We are trying to gauge from our users
> now to see if something is still lingering out there.
>
>
> Carissa Klemmer
> NCEP Central Operations
> Dataflow Team Lead
> 301-683-3835
>
> On Mon, Feb 22, 2016 at 8:52 AM, Patrick L. Francis <wxprofessor@xxxxxxxxx
> > wrote:
>
>>
>>
>> All,
>>
>> Something bigger than just CONDUIT users occurred on the 19th. We had
>> numerous users of other servers complain of similar drops in transfer
>> rates. A few of them were also able to move downloads to a new location
>> where speeds were normal. We believe that this issue was outside of NCEP
>> due to that reasoning. Having said that though, is anyone still seeing
>> abnormal rates?
>>
>>
>>
>>
>>
>> Hi Carissa J
>>
>>
>>
>> From our Amazon location in VA, everything is fine as it was on Fridayâ
>> From our colo facility, packet loss is still shown via 140.90.111.36, which
>> is the hop from gigapop to ncep.. here is an MTR showing the packet loss:
>>
>>
>>
>> http://drmalachi.org/files/ncep/he-ncep.2016.02.22.jpg
>>
>>
>>
>> Notice that since our colo is directly on the Hurricane Electric
>> backbone, there are only a few hops to get to the ncep server, and pings
>> only jump once they reach gigapop.
>>
>>
>>
>> Here is a screen cap of current download speeds for 0.25deg gfs from the
>> ncep server to our colo:
>>
>>
>>
>> http://drmalachi.org/files/ncep/he-ncep.wget.2016.02.22.jpg
>>
>>
>>
>> In order to ensure proper delivery, we have setup a push / pull from our
>> amazon box to the colo, but most people probably wonât have that
>> flexibility.. The issue appears to potentially be related to
>> packetfiltering and / or redirection / funneling via certain routes, but
>> thatâs just a guess J
>>
>>
>>
>> Happy Monday J
>>
>>
>>
>> Cheers,
>>
>>
>>
>> --patrick
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> -------------------------------------------------------
>>
>> Patrick L. Francis
>>
>> Vice President of Research & Development
>>
>>
>>
>> Aeris Weather
>>
>>
>>
>> http://aerisweather.com/
>>
>> http://modelweather.com/
>>
>>
>>
>> http://facebook.com/wxprofessor/
>>
>>
>>
>> --------------------------------------------------------
>>
>
>
> _______________________________________________
> conduit mailing list
> conduit@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>



-- 
âThe presence of those seeking the truth is infinitely to be preferred to
the presence of those who think theyâve found it."
 -- Terry Prachett

-------------------------------------------
Karen.Cooper@xxxxxxxx

Phone#:  405-325-6456
Cell:   405-834-8559
National Severe Storms Laboratory