Dear Anne and all,
Just a note to let you know we are still experiencing the high latencies. In
fact, today the latencies from both vm-lnx-conduit1 and vm-lnx-conduit2 are
high.
Pete
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu
[cid:80d659bc-50dd-4c2f-86f6-1f5503d4bd43]
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
[cid:89b269bb-9c86-4f34-8e2c-491d8648b7c9]
<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - poker@xxxxxxxxxxxx
________________________________
From: Anne Myckow - NOAA Federal <anne.myckow@xxxxxxxx>
Sent: Friday, August 20, 2021 12:14 PM
To: Pete Pokrandt <poker@xxxxxxxxxxxx>
Cc: dustin.sheffler@xxxxxxxx <dustin.sheffler@xxxxxxxx>;
ncep.list.pmb-dataflow@xxxxxxxx <ncep.list.pmb-dataflow@xxxxxxxx>;
conduit@xxxxxxxxxxxxxxxx <conduit@xxxxxxxxxxxxxxxx>;
support-conduit@xxxxxxxxxxxxxxxx <support-conduit@xxxxxxxxxxxxxxxx>
Subject: Re: High CONDUIT latencies from vm-lnx-conduit2.ncep.noaa.gov
Pete,
conduit.ncep.noaa.gov<http://conduit.ncep.noaa.gov> is a load-balanced DNS that
points to both conduit1 and conduit2 servers on the backend. I'm going to see
if we can push you all off of conduit2 for now, hopefully those of you
connected to conduit2 will see a brief interruption and then connect to
conduit1 automatically.
More to come.
Anne
On Fri, Aug 20, 2021 at 1:12 PM Pete Pokrandt
<poker@xxxxxxxxxxxx<mailto:poker@xxxxxxxxxxxx>> wrote:
It looks like conduit.ncep.noaa.gov<http://conduit.ncep.noaa.gov> is pulling
data from both vm-lnx-conduit1 and vm-linux-conduit2 - conduit1 seems ok, it's
just conduit2 that is showing the large lags.
I don't know how things are set up exactly, but it might work to have
conduit.ncep.noaa.gov<http://conduit.ncep.noaa.gov> only request CONDUIT data
from vm-lnx-conduit1 until the problem with feeding from conduit2 is resolved?
Unidata folks, any suggestions from your end?
Thanks, we do appreciate all your work on our behalf!
Pete
<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - poker@xxxxxxxxxxxx<mailto:poker@xxxxxxxxxxxx>
________________________________
From: Anne Myckow - NOAA Federal
<anne.myckow@xxxxxxxx<mailto:anne.myckow@xxxxxxxx>>
Sent: Friday, August 20, 2021 12:07 PM
To: Pete Pokrandt <poker@xxxxxxxxxxxx<mailto:poker@xxxxxxxxxxxx>>
Cc: dustin.sheffler@xxxxxxxx<mailto:dustin.sheffler@xxxxxxxx>
<dustin.sheffler@xxxxxxxx<mailto:dustin.sheffler@xxxxxxxx>>;
ncep.list.pmb-dataflow@xxxxxxxx<mailto:ncep.list.pmb-dataflow@xxxxxxxx>
<ncep.list.pmb-dataflow@xxxxxxxx<mailto:ncep.list.pmb-dataflow@xxxxxxxx>>;
conduit@xxxxxxxxxxxxxxxx<mailto:conduit@xxxxxxxxxxxxxxxx>
<conduit@xxxxxxxxxxxxxxxx<mailto:conduit@xxxxxxxxxxxxxxxx>>;
support-conduit@xxxxxxxxxxxxxxxx<mailto:support-conduit@xxxxxxxxxxxxxxxx>
<support-conduit@xxxxxxxxxxxxxxxx<mailto:support-conduit@xxxxxxxxxxxxxxxx>>
Subject: Re: High CONDUIT latencies from
vm-lnx-conduit2.ncep.noaa.gov<http://vm-lnx-conduit2.ncep.noaa.gov>
Hi Pete,
We have a lot of systems and applications running out of College Park right
now, which I think is part of it. But I will have someone take a look at
conduit2 today, see if maybe we need to try and move your connections to
conduit1 instead.
Thanks,
Anne
On Fri, Aug 20, 2021 at 12:54 PM Pete Pokrandt
<poker@xxxxxxxxxxxx<mailto:poker@xxxxxxxxxxxx>> wrote:
Dear Anne, Dustin and all,
Did you see this? We are still experiencing high latencies of 800-1000 seconds
on our CONDUIT feeds during the times when the GFS comes through that appear to
be coming from the host
vm-lnx-conduit2.ncep.noaa.gov<http://vm-lnx-conduit2.ncep.noaa.gov>
Here are the most recent lags. Any ideas?
Thanks,
Pete
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu
[cid:17b648d8092cb971f163]
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
[cid:17b648d8092cb971f164]
<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - poker@xxxxxxxxxxxx<mailto:poker@xxxxxxxxxxxx>
________________________________
From: Pete Pokrandt
Sent: Wednesday, August 18, 2021 3:02 PM
To: anne.myckow@xxxxxxxx<mailto:anne.myckow@xxxxxxxx>
<anne.myckow@xxxxxxxx<mailto:anne.myckow@xxxxxxxx>>;
dustin.sheffler@xxxxxxxx<mailto:dustin.sheffler@xxxxxxxx>
<dustin.sheffler@xxxxxxxx<mailto:dustin.sheffler@xxxxxxxx>>;
ncep.list.pmb-dataflow@xxxxxxxx<mailto:ncep.list.pmb-dataflow@xxxxxxxx>
<ncep.list.pmb-dataflow@xxxxxxxx<mailto:ncep.list.pmb-dataflow@xxxxxxxx>>
Cc: conduit@xxxxxxxxxxxxxxxx<mailto:conduit@xxxxxxxxxxxxxxxx>
<conduit@xxxxxxxxxxxxxxxx<mailto:conduit@xxxxxxxxxxxxxxxx>>;
support-conduit@xxxxxxxxxxxxxxxx<mailto:support-conduit@xxxxxxxxxxxxxxxx>
<support-conduit@xxxxxxxxxxxxxxxx<mailto:support-conduit@xxxxxxxxxxxxxxxx>>
Subject: High CONDUIT latencies from
vm-lnx-conduit2.ncep.noaa.gov<http://vm-lnx-conduit2.ncep.noaa.gov>
Dear Anne, Dustin and all,
Recently we have noticed fairly high latencies on the CONDUIT ldm data feed
originating from the machine
vm-lnx-conduit2.ncep.noaa.gov<http://vm-lnx-conduit2.ncep.noaa.gov>. The feed
originating from
vm-lnx-conduit1.ncep.noaa.gov<http://vm-lnx-conduit1.ncep.noaa.gov> does not
have the high latencies. Unidata and other top level feeds are seeing similar
high latencies from
vm-lnx-conduit2.ncep.noaa.gov<http://vm-lnx-conduit2.ncep.noaa.gov>.
Here are some graphs showing the latencies that I'm seeing:
From
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu
- latencies for CONDUIT data arriving at our UW-Madison AOS ingest machine
[cid:17b648d8091cb971f161]
From
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?conduit.unidata.ucar.edu
(latencies at Unidata)
[cid:17b648d8092cb971f162]
At least here at UW-Madison, these latencies are causing us to lose some data
during the large GFS/GEFS periods.
Any idea what might be causing this?
Pete
<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - poker@xxxxxxxxxxxx<mailto:poker@xxxxxxxxxxxx>
--
Anne Myckow
Dataflow Team Lead
NWS/NCEP/NCO
--
Anne Myckow
Dataflow Team Lead
NWS/NCEP/NCO
