Re: [conduit] Large CONDUIT lag from some top levels began after 12 UTC May 16

  • To: Pete Pokrandt <poker@xxxxxxxxxxxx>
  • Subject: Re: [conduit] Large CONDUIT lag from some top levels began after 12 UTC May 16
  • From: Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>
  • Date: Mon, 18 May 2015 13:19:46 -0400
Pete,

Starting May 16th both UCAR and GSD have also reported awful transfer rates
to many of our servers out here. The timing lines up that I am suspicious
that you are caught in whatever path is also affecting them from Boulder.
We have engaged the NOAA campus to investigate the possible issue.

Carissa Klemmer
NCEP Central Operations
Production Management Branch Dataflow Team
301-683-3835

On Mon, May 18, 2015 at 12:21 PM, Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:

> Just a heads-up/curiosity - did something change on one of the top level
> CONDUIT servers after the 12 UTC run on May 16, 2015?
>
> The attached gif, although it does not have times on the bottom, was
> grabbed around 01 UTC May 18th. It shows my lag from idd.aos.wisc.edu to
> the two top level CONDUIT servers (conduit.ncep.noaa.gov and
> ncepldm4.woc.noaa.gov)
>
> Time on the X axis is positive to the right, and starts about 2 days ago
> on the left, ending at the picture time on the right.
>
> Each of the 8 clusters of data along the X axis is one of the 6 hourly
> model cycles, so the farthest right one would be the 18 UTC 17 run cycle.
> Working to the left, you get 12, 06, 00, and 18 UTC 16 runs where the lag
> from the machines represented in blue and green got really large (2000-3000
> seconds) all of a sudden. The previous three cycles (12, 06, and 00 UTC May
> 16) have lags no larger than ~30-60 seconds, as they have been for the past
> several weeks.
>
> I don't appear to be losing any data because of this, but if something did
> change, I wanted to point out that I noticed this sudden increase in lag
> times. Looking at graphs from a few selected other conduit sites, some show
> a similar issue(eg atm.ucar.edu)  and others don't seem to (
> cyclone.plymouth.edu), so maybe it's a routing change?
>
> I did just talk with Jerry Robaidek at SSEC here at UW, and he told me
> that they started experiencing much slower data rates on their pull of data
> from the Japanese Himawari satellite around 20 UTC on Saturday May 16
> also.. Interesting.. Points to maybe a routing issue somewhere?
>
> Pete
>
>
> --
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086  - poker@xxxxxxxxxxxx
>
>