Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - started a week or so ago

  • To: Anne Myckow - NOAA Affiliate <anne.myckow@xxxxxxxx>
  • Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - started a week or so ago
  • From: Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
  • Date: Wed, 3 Apr 2019 15:07:14 -0500
Hello Anne,

I'll jump in here as well. Consider the CONDUIT delays at UNIDATA:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu


And now, Wisconsin:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu


And finally, the University of Washington:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+freshair1.atmos.washington.edu


All three of whom have direct feeds from you. Flipping over to Boulder
definitely caused a major improvement. There was still a brief spike in
delay, but much shorter and minimal
compared to what it was.

Gilbert

On Wed, Apr 3, 2019 at 10:03 AM Anne Myckow - NOAA Affiliate <
anne.myckow@xxxxxxxx> wrote:

> Hi Pete,
>
> As of yesterday we failed almost all of our applications to our site in
> Boulder (meaning away from CONDUIT). Have you noticed an improvement in
> your speeds since yesterday afternoon? If so this will give us a clue that
> maybe there's something interfering on our side that isn't specifically
> CONDUIT, but another app that might be causing congestion. (And if it's the
> same then that's a clue in the other direction.)
>
> Thanks,
> Anne
>
> On Mon, Apr 1, 2019 at 3:24 PM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>
>> The lag here at UW-Madison was up to 1200 seconds today, and that's with
>> a 10-way split feed. Whatever is causing the issue has definitely not been
>> resolved, and historically is worse during the work week than on the
>> weekends. If that helps at all.
>>
>> Pete
>>
>>
>>
>> <http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>
>> --
>> Pete Pokrandt - Systems Programmer
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>> 608-262-3086  - poker@xxxxxxxxxxxx
>>
>> ------------------------------
>> *From:* Anne Myckow - NOAA Affiliate <anne.myckow@xxxxxxxx>
>> *Sent:* Thursday, March 28, 2019 4:28 PM
>> *To:* Person, Arthur A.
>> *Cc:* Carissa Klemmer - NOAA Federal; Pete Pokrandt;
>> _NCEP.List.pmb-dataflow; conduit@xxxxxxxxxxxxxxxx;
>> support-conduit@xxxxxxxxxxxxxxxx
>> *Subject:* Re: [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed -
>> started a week or so ago
>>
>> Hello Art,
>>
>> We will not be upgrading to version 6.13 on these systems as they are not
>> robust enough to support the local logging inherent in the new version.
>>
>> I will check in with my team on if there are any further actions we can
>> take to try and troubleshoot this issue, but I fear we may be at the limit
>> of our ability to make this better.
>>
>> Iâll let you know tomorrow where we stand. Thanks.
>> Anne
>>
>> On Mon, Mar 25, 2019 at 3:00 PM Person, Arthur A. <aap1@xxxxxxx> wrote:
>>
>> Carissa,
>>
>>
>> Can you report any status on this inquiry?
>>
>>
>> Thanks...          Art
>>
>>
>> Arthur A. Person
>> Assistant Research Professor, System Administrator
>> Penn State Department of Meteorology and Atmospheric Science
>> email:  aap1@xxxxxxx, phone:  814-863-1563 <callto:814-863-1563>
>>
>>
>> ------------------------------
>> *From:* Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>
>> *Sent:* Tuesday, March 12, 2019 8:30 AM
>> *To:* Pete Pokrandt
>> *Cc:* Person, Arthur A.; conduit@xxxxxxxxxxxxxxxx;
>> support-conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow
>> *Subject:* Re: Large lags on CONDUIT feed - started a week or so ago
>>
>> Hi Everyone
>>
>> Iâve added the Dataflow team email to the thread. I havenât heard that
>> any changes were made or that any issues were found. But the team can look
>> today and see if we have any signifiers of overall slowness with anything.
>>
>> Dataflow, try taking a look at the new Citrix or VM troubleshooting tools
>> if there are any abnormal signatures that may explain this.
>>
>> On Monday, March 11, 2019, Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>
>> Art,
>>
>> I don't know if NCEP ever figured anything out, but I've been able to
>> keep my latencies reasonable (300-600s max, mostly during the 12 UTC model
>> suite) by splitting my CONDUIT request 10 ways, instead of the 5 that I had
>> been doing, or in a single request. Maybe give that a try and see if it
>> helps at all.
>>
>> Pete
>>
>>
>>
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133526700&sdata=rISFh7zNHrOY%2B4K4qBEqwtB2zq%2B0IE4l%2FmqYXRAcM6A%3D&reserved=0>
>> --
>> Pete Pokrandt - Systems Programmer
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>> 608-262-3086  - poker@xxxxxxxxxxxx
>>
>> ------------------------------
>> *From:* Person, Arthur A. <aap1@xxxxxxx>
>> *Sent:* Monday, March 11, 2019 3:45 PM
>> *To:* Holly Uhlenhake - NOAA Federal; Pete Pokrandt
>> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>> *Subject:* Re: [conduit] Large lags on CONDUIT feed - started a week or
>> so ago
>>
>>
>> Holly,
>>
>>
>> Was there any resolution to this on the NCEP end?  I'm still seeing
>> terrible delays (1000-4000 seconds) receiving data from
>> conduit.ncep.noaa.gov
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fconduit.ncep.noaa.gov&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133536705&sdata=5DIx0hfLu7Q1FxggAkcsn3845ayF0rfUUbnAADYjblY%3D&reserved=0>.
>> It would be helpful to know if things are resolved at NCEP's end so I know
>> whether to look further down the line.
>>
>>
>> Thanks...           Art
>>
>>
>> Arthur A. Person
>> Assistant Research Professor, System Administrator
>> Penn State Department of Meteorology and Atmospheric Science
>> email:  aap1@xxxxxxx, phone:  814-863-1563 <callto:814-863-1563>
>>
>>
>> ------------------------------
>> *From:* conduit-bounces@xxxxxxxxxxxxxxxx <
>> conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Holly Uhlenhake - NOAA
>> Federal <holly.uhlenhake@xxxxxxxx>
>> *Sent:* Thursday, February 21, 2019 12:05 PM
>> *To:* Pete Pokrandt
>> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>> *Subject:* Re: [conduit] Large lags on CONDUIT feed - started a week or
>> so ago
>>
>> Hi Pete,
>>
>> We'll take a look and see if we can figure out what might be going on.
>> We haven't done anything to try and address this yet, but based on your
>> analysis I'm suspicious that it might be tied to a resource constraint on
>> the VM or the blade it resides on.
>>
>> Thanks,
>> Holly Uhlenhake
>> Acting Dataflow Team Lead
>>
>> On Thu, Feb 21, 2019 at 11:32 AM Pete Pokrandt <poker@xxxxxxxxxxxx>
>> wrote:
>>
>> Just FYI, data is flowing, but the large lags continue.
>>
>>
>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bidd.aos.wisc.edu&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133536705&sdata=KqSHAeB6XYUOR2tVzRRSdMCzVahKVPOiPzvC9xoHMBU%3D&reserved=0>
>>
>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bconduit.unidata.ucar.edu&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133546714&sdata=I1mvCqaacew8DEknYPh3%2Fd69L6K2i6CtIKpgHDz9GT8%3D&reserved=0>
>>
>> Pete
>>
>>
>>
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133546714&sdata=pvl82se6O%2Fvtzi4DBKM%2F0YrJALvDoPE4k3Nz9hSUyyk%3D&reserved=0>
>> --
>> Pete Pokrandt - Systems Programmer
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>> 608-262-3086  - poker@xxxxxxxxxxxx
>>
>> ------------------------------
>> *From:* conduit-bounces@xxxxxxxxxxxxxxxx <
>> conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt <
>> poker@xxxxxxxxxxxx>
>> *Sent:* Wednesday, February 20, 2019 12:07 PM
>> *To:* Carissa Klemmer - NOAA Federal
>> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>> *Subject:* Re: [conduit] Large lags on CONDUIT feed - started a week or
>> so ago
>>
>> Data is flowing again - picked up somewhere in the GEFS. Maybe CONDUIT
>> server was restarted, or ldm on it? Lags are large (3000s+) but dropping
>> slowly
>>
>> Pete
>>
>>
>>
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133556719&sdata=zcmqQMKWWa3YXEiNQyWYJFCPWCMFrR1kDC%2B8jMZgTuY%3D&reserved=0>
>> --
>> Pete Pokrandt - Systems Programmer
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>> 608-262-3086  - poker@xxxxxxxxxxxx
>>
>> ------------------------------
>> *From:* conduit-bounces@xxxxxxxxxxxxxxxx <
>> conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt <
>> poker@xxxxxxxxxxxx>
>> *Sent:* Wednesday, February 20, 2019 11:56 AM
>> *To:* Carissa Klemmer - NOAA Federal
>> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>> *Subject:* Re: [conduit] Large lags on CONDUIT feed - started a week or
>> so ago
>>
>> Just a quick follow-up - we started falling far enough behind (3600+ sec)
>> that we are losing data. We got short files starting at 174h into the GFS
>> run, and only got (incomplete) data through 207h.
>>
>> We have now not received any data on CONDUIT since 11:27 AM CST (1727
>> UTC) today (Wed Feb 20)
>>
>> Pete
>>
>>
>>
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133556719&sdata=zcmqQMKWWa3YXEiNQyWYJFCPWCMFrR1kDC%2B8jMZgTuY%3D&reserved=0>
>> --
>> Pete Pokrandt - Systems Programmer
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>> 608-262-3086  - poker@xxxxxxxxxxxx
>>
>> ------------------------------
>> *From:* conduit-bounces@xxxxxxxxxxxxxxxx <
>> conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt <
>> poker@xxxxxxxxxxxx>
>> *Sent:* Wednesday, February 20, 2019 11:28 AM
>> *To:* Carissa Klemmer - NOAA Federal
>> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>> *Subject:* [conduit] Large lags on CONDUIT feed - started a week or so
>> ago
>>
>> Carissa,
>>
>> We have been feeding CONDUIT using a 5 way split feed direct from
>> conduit.ncep.noaa.gov
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fconduit.ncep.noaa.gov&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133566724&sdata=zeEqXgG81kDQIcbzVlj78FXdyz2ld1LKhmVaYP9bA1M%3D&reserved=0>,
>> and it had been really good for some time, lags 30-60 seconds or less.
>>
>> However, the past week or so, we've been seeing some very large lags
>> during each 6 hour model suite - Unidata is also seeing these - they are
>> also feeding direct from conduit.ncep.noaa.gov
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fconduit.ncep.noaa.gov&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133576734&sdata=yPlWPGkRjNuMQUHK6ZMz2CSOnt5GnyP0N5WBJ2Xl0n4%3D&reserved=0>
>> .
>>
>>
>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bidd.aos.wisc.edu&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133576734&sdata=g%2FWWKnNIJNPT4MRavn0YLNAUlKVBFl49%2Beipn5ftYkw%3D&reserved=0>
>>
>>
>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bconduit.unidata.ucar.edu&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133586734&sdata=p%2F1GcWu%2FAJUQwg1qWexK5DgSQCEjUn1T2udhtx4fix8%3D&reserved=0>
>>
>>
>> Any idea what's going on, or how we can find out?
>>
>> Thanks!
>> Pete
>>
>>
>>
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133596744&sdata=eFcuheAVvxEYMwwGes9fTTPts9sdGjFA2pQ0rpcDAdw%3D&reserved=0>
>> --
>> Pete Pokrandt - Systems Programmer
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>> 608-262-3086  - poker@xxxxxxxxxxxx
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web.  Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> conduit mailing list
>> conduit@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133596744&sdata=kJgg9477GhOS4LXezS9HI6%2B%2BbkhqVgKhmtBmBXddoK4%3D&reserved=0>
>>
>>
>>
>> --
>> Carissa Klemmer
>> NCEP Central Operations
>> IDSB Branch Chief
>> 301-683-3835
>>
>> _______________________________________________
>> Ncep.list.pmb-dataflow mailing list
>> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>>
>> --
>> Anne Myckow
>> Lead Dataflow Analyst
>> NOAA/NCEP/NCO
>> 301-683-3825
>>
>
>
> --
> Anne Myckow
> Lead Dataflow Analyst
> NOAA/NCEP/NCO
> 301-683-3825
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> conduit mailing list
> conduit@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/



-- 
----

Gilbert Sebenste
Consulting Meteorologist
AllisonHouse, LLC