Re: High CONDUIT latencies from vm-lnx-conduit2.ncep.noaa.gov

Also, I'd like to know if there are any of you all that are *not* experiencing latency. Please let me know if you are in that camp.

Thanks so much,

Anne

On Wed, Aug 25, 2021 at 9:04 AM Anne Myckow - NOAA Federal <address@hidden> wrote:

Morning,

I don't see the crazy latency from that one cycle yesterday but it still looks pretty bad to me - do you concur?

Thanks,

Anne

On Tue, Aug 24, 2021 at 4:03 PM Anne Myckow - NOAA Federal <address@hidden> wrote:

Hi everyone,

We've tried rebooting the systems, I checked your graph and it looks like we won't know for a few cycles if it's better - can you let us know if you see something before we check it tomorrow morning?

Thanks,

Anne

On Tue, Aug 24, 2021 at 1:59 PM Tyle, Kevin R <address@hidden> wrote:

Hi all,

 

I can state that our GFS grib file reception via LDM has been extremely spotty, particularly for the F48-F192 forecast hour periods, for several weeks now. We feed from Pete’s LDM at UW-MSN so this is consistent with what Pete has been seeing.

 

It would be really nice if NCEP’s CONDUIT feed can return to the level of consistent service that we in the community had been accustomed to for many years.

 

Cheers,

 

Kevin

 

_________________________________________________

 

Kevin Tyle, M.S.; Manager of Departmental Computing

NSF XSEDE Campus Champion
Dept. of Atmospheric & Environmental Sciences  
UAlbany ETEC Bldg – Harriman Campus
1220 Washington Avenue, Room 419                       
Albany, NY 12222
address@hidden | 518-442-4578 | @nywxguy | he/him/his        

_________________________________________________

 

From: conduit <address@hidden> On Behalf Of Pete Pokrandt via conduit
Sent: Tuesday, August 24, 2021 1:26 PM
To: Anne Myckow - NOAA Federal <address@hidden>
Cc: address@hidden; address@hidden; address@hidden; address@hidden
Subject: Re: [conduit] High CONDUIT latencies from vm-lnx-conduit2.ncep.noaa.gov

 

Dear Anne and all,

 

Just a note to let you know we are still experiencing the high latencies. In fact, today the latencies from both vm-lnx-conduit1 and vm-lnx-conduit2 are high.

 

Pete

 

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu

 

 

 

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu

 

-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden

 

---

From: Anne Myckow - NOAA Federal <address@hidden>
Sent: Friday, August 20, 2021 12:14 PM
To: Pete Pokrandt <address@hidden>
Cc: address@hidden <address@hidden>; address@hidden <address@hidden>; address@hidden <address@hidden>; address@hidden <address@hidden>
Subject: Re: High CONDUIT latencies from vm-lnx-conduit2.ncep.noaa.gov

 

Pete,

 

conduit.ncep.noaa.gov is a load-balanced DNS that points to both conduit1 and conduit2 servers on the backend. I'm going to see if we can push you all off of conduit2 for now, hopefully those of you connected to conduit2 will see a brief interruption and then connect to conduit1 automatically.

 

More to come.

Anne

 

On Fri, Aug 20, 2021 at 1:12 PM Pete Pokrandt <address@hidden> wrote:

It looks like conduit.ncep.noaa.gov is pulling data from both vm-lnx-conduit1 and vm-linux-conduit2 - conduit1 seems ok, it's just conduit2 that is showing the large lags.

 

I don't know how things are set up exactly, but it might work to have conduit.ncep.noaa.gov only request CONDUIT data from vm-lnx-conduit1 until the problem with feeding from conduit2 is resolved? 

Unidata folks, any suggestions from your end?

Thanks, we do appreciate all your work on our behalf!

Pete

 

 

-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden

 

---

From: Anne Myckow - NOAA Federal <address@hidden>
Sent: Friday, August 20, 2021 12:07 PM
To: Pete Pokrandt <address@hidden>
Cc: address@hidden <address@hidden>; address@hidden <address@hidden>; address@hidden <address@hidden>; address@hidden <address@hidden>
Subject: Re: High CONDUIT latencies from vm-lnx-conduit2.ncep.noaa.gov

 

Hi Pete,

 

We have a lot of systems and applications running out of College Park right now, which I think is part of it. But I will have someone take a look at conduit2 today, see if maybe we need to try and move your connections to conduit1 instead.

 

Thanks,

Anne

 

On Fri, Aug 20, 2021 at 12:54 PM Pete Pokrandt <address@hidden> wrote:

Dear Anne, Dustin and all,

 

Did you see this? We are still experiencing high latencies of 800-1000 seconds on our CONDUIT feeds during the times when the GFS comes through that appear to be coming from the host

 

vm-lnx-conduit2.ncep.noaa.gov

 

Here are the most recent lags. Any ideas?

Thanks,

Pete

 

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu

 

 

 

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu

 

 

-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden

 

---

From: Pete Pokrandt
Sent: Wednesday, August 18, 2021 3:02 PM
To: address@hidden <address@hidden>; address@hidden <address@hidden>; address@hidden <address@hidden>
Cc: address@hidden <address@hidden>; address@hidden <address@hidden>
Subject: High CONDUIT latencies from vm-lnx-conduit2.ncep.noaa.gov

 

Dear Anne, Dustin and all,

 

Recently we have noticed fairly high latencies on the CONDUIT ldm data feed originating from the machine vm-lnx-conduit2.ncep.noaa.gov. The feed originating from vm-lnx-conduit1.ncep.noaa.gov does not have the high latencies. Unidata and other top level feeds are seeing similar high latencies from vm-lnx-conduit2.ncep.noaa.gov.

 

Here are some graphs showing the latencies that I'm seeing:

 

From https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu - latencies for CONDUIT data arriving at our UW-Madison AOS ingest machine

 

 

From https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?conduit.unidata.ucar.edu (latencies at Unidata)

 

 

At least here at UW-Madison, these latencies are causing us to lose some data during the large GFS/GEFS periods.

 

Any idea what might be causing this?

 

Pete

 

 

 

 

-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - address@hidden

 

--

Anne Myckow

Dataflow Team Lead

NWS/NCEP/NCO

 

--

Anne Myckow

Dataflow Team Lead

NWS/NCEP/NCO

--

Anne Myckow

Dataflow Team Lead

NWS/NCEP/NCO

--

Anne Myckow

Dataflow Team Lead

NWS/NCEP/NCO

--

Anne Myckow

Dataflow Team Lead

NWS/NCEP/NCO