[conduit] Large CONDUIT lag from some top levels began after 12 UTC May 16

  • To: Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>
  • Subject: [conduit] Large CONDUIT lag from some top levels began after 12 UTC May 16
  • From: Pete Pokrandt <poker@xxxxxxxxxxxx>
  • Date: Mon, 18 May 2015 11:21:49 -0500
Just a heads-up/curiosity - did something change on one of the top level CONDUIT servers after the 12 UTC run on May 16, 2015?

The attached gif, although it does not have times on the bottom, was grabbed around 01 UTC May 18th. It shows my lag from idd.aos.wisc.edu to the two top level CONDUIT servers (conduit.ncep.noaa.gov and ncepldm4.woc.noaa.gov)

Time on the X axis is positive to the right, and starts about 2 days ago on the left, ending at the picture time on the right.

Each of the 8 clusters of data along the X axis is one of the 6 hourly model cycles, so the farthest right one would be the 18 UTC 17 run cycle. Working to the left, you get 12, 06, 00, and 18 UTC 16 runs where the lag from the machines represented in blue and green got really large (2000-3000 seconds) all of a sudden. The previous three cycles (12, 06, and 00 UTC May 16) have lags no larger than ~30-60 seconds, as they have been for the past several weeks.

I don't appear to be losing any data because of this, but if something did change, I wanted to point out that I noticed this sudden increase in lag times. Looking at graphs from a few selected other conduit sites, some show a similar issue(eg atm.ucar.edu) and others don't seem to (cyclone.plymouth.edu), so maybe it's a routing change?

I did just talk with Jerry Robaidek at SSEC here at UW, and he told me that they started experiencing much slower data rates on their pull of data from the Japanese Himawari satellite around 20 UTC on Saturday May 16 also.. Interesting.. Points to maybe a routing issue somewhere?

Pete


--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - poker@xxxxxxxxxxxx

Attachment: iddstats_nc_idd_aos_wisc_edu_conduit_01UTC_May_18_2015_8PM_CDT_May_17_2015.gif
Description: GIF image