Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - started a week or so ago

  • To: "Tyle, Kevin R" <ktyle@xxxxxxxxxx>
  • Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - started a week or so ago
  • From: Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
  • Date: Tue, 16 Apr 2019 14:44:07 -0500
<html><head><meta http-equiv="content-type" content="text/html; 
charset=utf-8"></head><body dir="auto"><div dir="ltr"></div><div dir="ltr">Yes, 
here at AllisonHouse too...we can feed from a number of sites, and all of them 
were dropping GFS, and delayed by an hour.</div><div dir="ltr"><br></div><div 
dir="ltr">Gilbert</div><div dir="ltr"><br>On Apr 16, 2019, at 2:39 PM, Tyle, 
Kevin R &lt;<a href="mailto:ktyle@xxxxxxxxxx";>ktyle@xxxxxxxxxx</a>&gt; 
wrote:<br><br></div><blockquote type="cite"><div dir="ltr">

<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">



<div style="font-family: Tahoma, Geneva, sans-serif; font-size: 12pt; color: 
rgb(0, 0, 0);">
For what it's worth, our 12Z GFS data ingest was quite bad today ... many lost 
products beyond F168 (we feed from UWisc-MSN primary and PSU secondary).</div>
<div style="font-family: Tahoma, Geneva, sans-serif; font-size: 12pt; color: 
rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; 
background-color:#FFFFFF; font-family:Tahoma,Geneva,sans-serif">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">_____________________________________________<br>
Kevin Tyle, M.S.;&nbsp;Manager of Departmental Computing<br>
Dept. of Atmospheric &amp; Environmental Sciences&nbsp;&nbsp; <br>
University at Albany<br>
Earth Science 235, 1400 Washington 
Avenue&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <br>
Albany, NY 12222<br>
Email: <a href="mailto:ktyle@xxxxxxxxxx";>ktyle@xxxxxxxxxx</a><br>
Phone: 
518-442-4578&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <br>
_____________________________________________<br>
</div>
</span></font></div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" 
style="font-size:11pt" color="#000000"><b>From:</b> <a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx";>conduit-bounces@xxxxxxxxxxxxxxxx</a>
 &lt;<a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx";>conduit-bounces@xxxxxxxxxxxxxxxx</a>&gt;
 on behalf of Pete Pokrandt &lt;<a 
href="mailto:poker@xxxxxxxxxxxx";>poker@xxxxxxxxxxxx</a>&gt;<br>
<b>Sent:</b> Tuesday, April 16, 2019 12:00 PM<br>
<b>To:</b> Dustin Sheffler - NOAA Federal; Mike Zuranski<br>
<b>Cc:</b> Kevin Goebbert; <a 
href="mailto:conduit@xxxxxxxxxxxxxxxx";>conduit@xxxxxxxxxxxxxxxx</a>; Derek 
VanPelt - NOAA Affiliate; _NCEP.List.pmb-dataflow; <a 
href="mailto:support-conduit@xxxxxxxxxxxxxxxx";>support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT 
feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
<style type="text/css" style="display:none">
<!--
p
        {margin-top:0;
        margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
All,</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
Just keeping this in the foreground.&nbsp;</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
CONDUIT lags continue to be very large compared to what they were previous to 
whatever changed back in February. Prior to that, we rarely saw lags more than 
~300s. Now they are routinely 1500-2000s at UW-Madison and Penn State, 
and&nbsp; over 3000s at Unidata -
 they appear to be on the edge of losing data. This does not bode well with all 
of the IDP applications failing back over to CP today..</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Can we send you some traceroutes and you back to us to maybe try to isolate 
where in the network this is happening? It feels like congestion or a bad route 
somewhere - the lags seem to be worse on weekdays than weekends if that helps 
at all.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Here are the current CONDUIT lags to UW-Madison, Penn State and Unidata.<br>
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<img size="13061" tabindex="-1" style="max-width:100%" 
data-outlook-trace="F:2|T:2" src="cid:d125d74f-ee3d-4519-8a8c-6521e49edbe9"><br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
&lt;iddstats_CONDUIT_idd_meteo_psu_edu_ending_20190416_1600UTC.gif&gt;<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<img size="22940" tabindex="-1" style="max-width:100%" 
data-outlook-trace="F:2|T:2" src="cid:657a7541-6bbe-4ad1-a141-0aba526feced"><br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div id="x_Signature">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; 
color:#000000; background-color:#FFFFFF; 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool";></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a 
href="mailto:poker@xxxxxxxxxxxx";>poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div id="x_appendonsend"></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" 
color="#000000" style="font-size:11pt"><b>From:</b> <a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx";>conduit-bounces@xxxxxxxxxxxxxxxx</a>
 &lt;<a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx";>conduit-bounces@xxxxxxxxxxxxxxxx</a>&gt;
 on behalf of Dustin Sheffler - NOAA Federal &lt;<a 
href="mailto:dustin.sheffler@xxxxxxxx";>dustin.sheffler@xxxxxxxx</a>&gt;<br>
<b>Sent:</b> Tuesday, April 9, 2019 12:52 PM<br>
<b>To:</b> Mike Zuranski<br>
<b>Cc:</b> <a 
href="mailto:conduit@xxxxxxxxxxxxxxxx";>conduit@xxxxxxxxxxxxxxxx</a>; 
_NCEP.List.pmb-dataflow; Derek VanPelt - NOAA Affiliate; <a 
href="mailto:support-conduit@xxxxxxxxxxxxxxxx";>support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT 
feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
<div>
<div dir="ltr">
<div>Hi Mike,</div>
<div><br>
</div>
<div>Thanks for the feedback on NOMADS. We recently found a slowness issue when 
NOMADS is running out of our Boulder data center that is being worked on by our 
teams now that NOMADS is live out of the College Park data center. It's hard 
sometimes to quantify
 whether slowness issues that are only being reported by a handful of users is 
a result of something wrong in our data center, a bad network path between a 
customer (possibly just from a particular region of the country) and our data 
center, a local issue on
 the customers' end, or any other reason that might cause slowness. <br>
</div>
<div><br>
</div>
<div>Conduit is <u>only</u> ever run from our College Park data center. It's 
slowness is not tied into the Boulder NOMADS issue, but it does seem to be at 
least a little bit tied to which of our data centers NOMADS is running out of. 
When NOMADS is in Boulder
 along with the majority of our other NCEP applications, the strain on the 
College Park data center is minimal and Conduit appears to be running better as 
a result. When NOMADS runs in College Park (as it has since late yesterday) 
there is more strain on the
 data center and Conduit appears (based on provided user graphs) to run a bit 
worse around peak model times as a result. These are just my observations and 
we are still investigating what may have changed that caused the Conduit 
latencies to appear in the first
 place so that we can resolve this potential constraint. <br>
</div>
<div><br>
</div>
<div>-Dustin<br>
</div>
</div>
<br>
<div class="x_x_gmail_quote">
<div dir="ltr" class="x_x_gmail_attr">On Tue, Apr 9, 2019 at 4:28 PM Mike 
Zuranski &lt;<a 
href="mailto:zuranski@xxxxxxxxxxxxxxx";>zuranski@xxxxxxxxxxxxxxx</a>&gt; 
wrote:<br>
</div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; 
border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hi everyone,
<div><br>
</div>
<div>I've avoided jumping into this conversation since I don't deal much with 
Conduit these days, but Derek just mentioned something that I do have some 
applicable feedback on...<br>
<br>
&gt;&nbsp;<span style="color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif; font-size:16px">Two items happened 
last night.&nbsp;&nbsp;</span><span style="color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif; font-size:16px">1. NOMADS was moved 
back to College Park...<br>
<br>
We get nearly all of our model data via NOMADS.&nbsp; When it switched to 
Boulder last week we saw a significant drop in download speeds, down to a 
couple hundred KB/s or slower.&nbsp; Starting last night, we're back to speeds 
on the order of MB/s or tens of MB/s.&nbsp; Switching
 back to College Park seems to confirm for me something about routing from 
Boulder was responsible.&nbsp; But again this was all on NOMADS, not sure if 
it's related to happenings on Conduit.</span></div>
<div><span style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; 
font-size:16px"><br>
</span></div>
<div><span style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; 
font-size:16px">When I noticed this last week I sent an email to
<a href="mailto:sdm@xxxxxxxx"; target="_blank">sdm@xxxxxxxx</a> including a 
traceroute taken at the time, let me know if you'd like me to find that and 
pass it along here or someplace else.</span></div>
<div><span style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; 
font-size:16px"><br>
</span></div>
<div><span style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; 
font-size:16px">-Mike<br>
</span>
<div>
<div dir="ltr" class="x_x_gmail-m_4641072781522989288gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div><br>
</div>
<div>======================</div>
Mike Zuranski
<div>Meteorology Support Analyst<br>
</div>
<div>College of DuPage - Nexlab</div>
<div><a href="http://Weather.cod.edu"; target="_blank">Weather.cod.edu</a></div>
<div>======================</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="x_x_gmail_quote">
<div dir="ltr" class="x_x_gmail_attr">On Tue, Apr 9, 2019 at 10:51 AM Person, 
Arthur A. &lt;<a href="mailto:aap1@xxxxxxx"; 
target="_blank">aap1@xxxxxxx</a>&gt; wrote:<br>
</div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; 
border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528divtagdefaultwrapper"
 dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px">Derek,</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Do we know what change might have 
been made around February 10th when the CONDUIT problems first started 
happening?&nbsp; Prior to that time, the CONDUIT feed had been very crisp for a 
long period of time.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Thanks...&nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; Art</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528Signature">
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528divtagdefaultwrapper"
 dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif,EmojiFont,&quot;Apple Color 
Emoji&quot;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Segoe UI 
Symbol&quot;,&quot;Android Emoji&quot;,EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528ms-rterangepaste-start"></span><span
 style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Arthur A.
 Person</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Assistant Research Professor, System 
Administrator</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Penn State Department of Meteorology and 
Atmospheric Science</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">email:</span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528Object" 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528OBJ_PREFIX_DWT1135_ZmEmailObjectHandler"
 style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><span 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528Object" 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528OBJ_PREFIX_DWT1137_ZmEmailObjectHandler"><a
 href="mailto:aap1@xxxxxxx"; target="_blank">aap1@xxxxxxx</a></span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">,
 phone:</span><span style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528Object" 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528OBJ_PREFIX_DWT1136_com_zimbra_phone"
 style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><a 
href="callto:814-863-1563" 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528LPNoLP" 
target="_blank" style="color:rgb(0,61,121)">814-863-1563</a></span><span 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528ms-rterangepaste-end"></span><br>
</p>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528divRplyFwdMsg" 
dir="ltr">
<font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b> Derek VanPelt - NOAA Affiliate &lt;<a 
href="mailto:derek.vanpelt@xxxxxxxx"; 
target="_blank">derek.vanpelt@xxxxxxxx</a>&gt;<br>
<b>Sent:</b> Tuesday, April 9, 2019 11:34 AM<br>
<b>To:</b> Holly Uhlenhake - NOAA Federal<br>
<b>Cc:</b> Carissa Klemmer - NOAA Federal; Person, Arthur A.; Pete Pokrandt; 
_NCEP.List.pmb-dataflow;
<a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] Large lags on CONDUIT feed - started a week or so 
ago</font>
<div>&nbsp;</div>
</div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>Hi all,</div>
<div><br>
</div>
<div>Two items happened last night.</div>
<div><br>
</div>
<div>1.&nbsp;&nbsp; NOMADS was moved back to College Park, which means there 
was a lot more traffic going out which will have effect on the Conduit 
latencies.&nbsp; We do not have a full load from the COllege Park Servers as 
many of the other applications are still running
 from Boulder, but NOMADS will certainly increase overall load.</div>
<div><br>
</div>
<div>2.&nbsp;&nbsp; As Holly said, there were further issues delaying and 
changing the timing of the model output yesterday afternoon/evening.&nbsp; I 
will be watching from our end, and monitoring the Unidata 48 hour graph (thank 
you for the link) throughout the day,
<br>
</div>
<div><br>
</div>
<div>Please let us know if you have questions or more information to help us 
analyse what you are seeing.&nbsp;
<br>
</div>
<div><br>
</div>
<div>Thank you,</div>
<div><br>
</div>
<div>Derek<br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<div>
<div dir="ltr"><br>
<div 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_gmail_quote">
<div dir="ltr" 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_gmail_attr">
On Tue, Apr 9, 2019 at 6:50 AM Holly Uhlenhake - NOAA Federal &lt;<a 
href="mailto:holly.uhlenhake@xxxxxxxx"; 
target="_blank">holly.uhlenhake@xxxxxxxx</a>&gt; wrote:<br>
</div>
<blockquote 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_gmail_quote" 
style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); 
padding-left:1ex">
<div dir="ltr">
<div>Hi Pete,</div>
<div><br>
</div>
<div>We also had an issue on the supercomputer yesterday where several models 
going to conduit would have been stacked on top of each other instead of coming 
out in a more spread out fashion.&nbsp; It's not inconceivable that conduit 
could have backed up working
 through the abnormally large glut of grib messages.&nbsp;&nbsp;&nbsp; Are 
things better this morning at all?<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Holly<br>
</div>
</div>
<br>
<div 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_gmail_quote">
<div dir="ltr" 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_gmail_attr">
On Tue, Apr 9, 2019 at 12:37 AM Pete Pokrandt &lt;<a 
href="mailto:poker@xxxxxxxxxxxx"; target="_blank">poker@xxxxxxxxxxxx</a>&gt; 
wrote:<br>
</div>
<blockquote 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_gmail_quote" 
style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); 
padding-left:1ex">
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Something changed starting with today's 18 UTC model cycle, and our lags shot 
up to over 3600 seconds, where we started losing data. They are growing again 
now with the 00 UTC cycle as well. PSU and Unidata CONDUIT stats show similar 
abnormally large lags.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
FYI.<br>
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474signature">
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474divtagdefaultwrapper"
 dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262069725&amp;sdata=ZmTVJ%2FCeylvBSnE4bfbvXoZEMi1qTxGIsV0ePOJmr0A%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474appendonsend">
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474divRplyFwdMsg"
 dir="ltr">
<font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b> Person, Arthur A. &lt;<a 
href="mailto:aap1@xxxxxxx"; target="_blank">aap1@xxxxxxx</a>&gt;<br>
<b>Sent:</b> Friday, April 5, 2019 2:10 PM<br>
<b>To:</b> Carissa Klemmer - NOAA Federal<br>
<b>Cc:</b> Pete Pokrandt; Derek VanPelt - NOAA Affiliate; Gilbert Sebenste; <a 
href="mailto:conduit@xxxxxxxxxxxxxxxx"; target="_blank">
conduit@xxxxxxxxxxxxxxxx</a>; _NCEP.List.pmb-dataflow; <a 
href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; target="_blank">
support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: Large lags on CONDUIT feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
<div dir="ltr">
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_divtagdefaultwrapper"
 dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px">Carissa,</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">The Boulder connection is 
definitely performing very well for CONDUIT.&nbsp; Although there have been a 
couple of little blips (~ 120 seconds) since yesterday, overall the performance 
is superb.&nbsp; I don't think it's quite
 as clean as prior to the ~February 10th date&nbsp;when the D.C. connection 
went bad, but it's still excellent performance.&nbsp; Here's our graph now with 
a single connection (no splits):</p>
<p style="margin-top:0px; margin-bottom:0px"><img size="26175" 
style="width:736px; max-width:100%" data-outlook-trace="F:2|T:2" 
src="cid:16a02dbca53f456b1e55"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">My next question is:&nbsp; Will 
CONDUIT stay pointing at Boulder until D.C. is fixed, or might you be required 
to switch back to D.C. at some point before that?</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Thanks...&nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Art</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_Signature">
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_divtagdefaultwrapper"
 dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif,EmojiFont,&quot;Apple Color 
Emoji&quot;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Segoe UI 
Symbol&quot;,&quot;Android Emoji&quot;,EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_ms-rterangepaste-start"></span><span
 style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Arthur
 A. Person</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Assistant Research Professor, System 
Administrator</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Penn State Department of Meteorology and 
Atmospheric Science</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">email:</span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_Object"
 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_OBJ_PREFIX_DWT1135_ZmEmailObjectHandler"
 style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><span 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_Object"
 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_OBJ_PREFIX_DWT1137_ZmEmailObjectHandler"><a
 href="mailto:aap1@xxxxxxx"; target="_blank">aap1@xxxxxxx</a></span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">,
 phone:</span><span style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_Object"
 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_OBJ_PREFIX_DWT1136_com_zimbra_phone"
 style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><a 
href="callto:814-863-1563" 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474LPNoLP"
 target="_blank" style="color:rgb(0,61,121)">814-863-1563</a></span><span 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_ms-rterangepaste-end"></span><br>
</p>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div 
id="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_divRplyFwdMsg"
 dir="ltr">
<font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b> Carissa Klemmer - NOAA Federal &lt;<a 
href="mailto:carissa.l.klemmer@xxxxxxxx"; 
target="_blank">carissa.l.klemmer@xxxxxxxx</a>&gt;<br>
<b>Sent:</b> Thursday, April 4, 2019 6:22 PM<br>
<b>To:</b> Person, Arthur A.<br>
<b>Cc:</b> Pete Pokrandt; Derek VanPelt - NOAA Affiliate; Gilbert Sebenste; <a 
href="mailto:conduit@xxxxxxxxxxxxxxxx"; target="_blank">
conduit@xxxxxxxxxxxxxxxx</a>; _NCEP.List.pmb-dataflow; <a 
href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; target="_blank">
support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: Large lags on CONDUIT feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
<div>Catching up here.
<div><br>
</div>
<div>Derek,</div>
<div>Do we have traceroutes from all users? Does anything in VCenter show any 
system resource constraints?<br>
<br>
On Thursday, April 4, 2019, Person, Arthur A. &lt;<a href="mailto:aap1@xxxxxxx"; 
target="_blank">aap1@xxxxxxx</a>&gt; wrote:<br>
<blockquote 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail-m_-1618515170744204033gmail-m_-7714654884662640474x_x_gmail_quote"
 style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); 
padding-left:1ex">
<div dir="ltr">
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px">Yeh, definitely looks "blipier" 
starting around 7Z this morning, but nothing like it was before.&nbsp; And all 
last night was clean.&nbsp; Here's our graph with a 2-way split, a huge 
improvement over what it was before the
 switch to Boulder:</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px"><img size="26389" 
style="width:736px; max-width:100%" data-outlook-trace="F:2|T:2" 
src="cid:16a02dbca53f456b1e53"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Agree with Pete that this 
morning's data probably isn't a good test since there were other factors.&nbsp; 
Since this seems so much better,&nbsp;I'm going to try switching to no split as 
an experiment and see how it holds up.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Art</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif,EmojiFont,&quot;Apple Color 
Emoji&quot;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Segoe UI 
Symbol&quot;,&quot;Android Emoji&quot;,EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Arthur A. Person</span><br 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Assistant Research Professor, System 
Administrator</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Penn State Department of Meteorology and 
Atmospheric Science</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">email:</span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><span><a 
href="mailto:aap1@xxxxxxx"; target="_blank">aap1@xxxxxxx</a></span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">,
 phone:</span><span style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><a 
href="callto:814-863-1563" target="_blank" 
style="color:rgb(0,61,121)">814-863-1563</a></span><span></span><br>
</p>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b> Pete Pokrandt &lt;<a 
href="mailto:poker@xxxxxxxxxxxx"; target="_blank">poker@xxxxxxxxxxxx</a>&gt;<br>
<b>Sent:</b> Thursday, April 4, 2019 1:51 PM<br>
<b>To:</b> Derek VanPelt - NOAA Affiliate<br>
<b>Cc:</b> Person, Arthur A.; Gilbert Sebenste; Anne Myckow - NOAA Affiliate; 
<a href="mailto:conduit@xxxxxxxxxxxxxxxx"; target="_blank">
conduit@xxxxxxxxxxxxxxxx</a>; _NCEP.List.pmb-dataflow; <a 
href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; target="_blank">
support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [Ncep.list.pmb-dataflow] [conduit] Large lags on CONDUIT 
feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Ah, so perhaps not a good test.. I'll set it back to a 5-way split and see how 
it looks tomorrow.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Thanks&nbsp;for the info,</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262079726&amp;sdata=nfgLZrEi9nKlsx20RpNyyHQVunPIH3Bwu5t2gMnhFYI%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b> Derek VanPelt - NOAA Affiliate &lt;<a 
href="mailto:derek.vanpelt@xxxxxxxx"; 
target="_blank">derek.vanpelt@xxxxxxxx</a>&gt;<br>
<b>Sent:</b> Thursday, April 4, 2019 12:38 PM<br>
<b>To:</b> Pete Pokrandt<br>
<b>Cc:</b> Person, Arthur A.; Gilbert Sebenste; Anne Myckow - NOAA Affiliate; 
<a href="mailto:conduit@xxxxxxxxxxxxxxxx"; target="_blank">
conduit@xxxxxxxxxxxxxxxx</a>; _NCEP.List.pmb-dataflow; <a 
href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; target="_blank">
support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [Ncep.list.pmb-dataflow] [conduit] Large lags on CONDUIT 
feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
<div>
<div dir="ltr">
<div>HI Pete -- we did have a separate issu hit the CONDUIT feed today.&nbsp; 
We should be recovering now, but the backlog was sizeable.&nbsp; If these 
numbers are not back to the baseline in the next hour or so please let us 
know.&nbsp; We are also watching our queues and
 they are decreasing, but not as quickly as we had hoped.<br>
</div>
<div><br>
</div>
<div>Thank you,</div>
<div><br>
</div>
<div>Derek<br>
</div>
<br>
<div>
<div dir="ltr">On Thu, Apr 4, 2019 at 1:26 PM 'Pete Pokrandt' via _NCEP 
list.pmb-dataflow &lt;<a href="mailto:ncep.list.pmb-dataflow@xxxxxxxx"; 
target="_blank">ncep.list.pmb-dataflow@xxxxxxxx</a>&gt; wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid 
rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
FYI - there is still a much larger lag for the 12 UTC run with a 5-way split 
compared to a 10-way split. It's better since everything else failed over to 
Boulder, but I'd venture to guess that's not the root of the problem.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<img size="10685" style="width:736px; max-width:100%" 
data-outlook-trace="F:2|T:2" src="cid:16a02dbca536351386f4"><br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Prior to whatever is going on to cause this, I don'r recall ever seeing lags 
this large with a 5-way split. It looked much more like the left hand side of 
this graph, with small increases in lag with each 6 hourly model run cycle, but 
more like 100 seconds
 vs the ~900 that I got this morning.<br>
<br>
FYI I am going to change back to a 10 way split for now.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262079726&amp;sdata=nfgLZrEi9nKlsx20RpNyyHQVunPIH3Bwu5t2gMnhFYI%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b>
<a href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a> &lt;<a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a>&gt; on behalf of Pete 
Pokrandt &lt;<a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a>&gt;<br>
<b>Sent:</b> Wednesday, April 3, 2019 4:57 PM<br>
<b>To:</b> Person, Arthur A.; Gilbert Sebenste; Anne Myckow - NOAA Affiliate<br>
<b>Cc:</b> <a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>; _NCEP.List.pmb-dataflow;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT 
feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Sorry, was out this morning and just had a chance to look into this. I concur 
with Art and Gilbert that things appear to have gotten better starting with the 
failover of everything else to Boulder yesterday. I will also reconfigure to go 
back to a 5-way split
 (as opposed to the 10-way split that I've been using since this issue began) 
and keep an eye on tomorrow's 12 UTC model run cycle - if the lags go up, it 
usually happens worst during that cycle, shortly before 18 UTC each 
day.&nbsp;</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
I'll report back tomorrow how it looks, or you can see at&nbsp;</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bidd.aos.wisc.edu&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262089730&amp;sdata=C0zwKdhnNi0Jpqteh%2B4yazfSjJHCj%2B9hSwE%2F1UOX29U%3D&amp;reserved=0";
 
target="_blank">http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu</a><br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262099739&amp;sdata=azCKUDriBN8O5WS3ytUMHYGjSEKvBQ483y%2FJPUjdIgQ%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b>
<a href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a> &lt;<a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a>&gt; on behalf of Person, 
Arthur A. &lt;<a href="mailto:aap1@xxxxxxx"; 
target="_blank">aap1@xxxxxxx</a>&gt;<br>
<b>Sent:</b> Wednesday, April 3, 2019 4:04 PM<br>
<b>To:</b> Gilbert Sebenste; Anne Myckow - NOAA Affiliate<br>
<b>Cc:</b> <a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>; _NCEP.List.pmb-dataflow;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT 
feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
<div dir="ltr">
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px">Anne,</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">I'll hop back in the loop here... 
for some reason these replies started going into my junk file (bleh).&nbsp; 
Anyway,&nbsp;<span style="font-size:12pt">I agree with Gilbert's 
assessment.&nbsp; Things turned real clean around 12Z
 yesterday, looking at the graphs.&nbsp; I usually look at <a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fflood.atmos.uiuc.edu&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262099739&amp;sdata=OEfEpm9%2BfxCxC5RQjY9wvKjsuLBaDymLq669BuMmyRE%3D&amp;reserved=0";
 target="_blank">
flood.atmos.uiuc.edu</a> when there are problem as their connection always 
seems to be the cleanest.&nbsp; If there are even small blips or ups and downs 
in their latencies, that usually means there's a network aberration somewhere 
that usually amplifies into hundreds
 or thousands of seconds at our site and elsewhere.&nbsp; Looking at their 
graph now, you can see the blipiness up until 12Z yesterday, and then it's flat 
(except for the one spike around 16Z today which I would ignore):</span></p>
<p style="margin-top:0px; margin-bottom:0px"><span style="font-size:12pt"><br>
</span></p>
<p style="margin-top:0px; margin-bottom:0px"><span 
style="font-size:12pt">&lt;pastedImage.png&gt;<br>
</span></p>
<p style="margin-top:0px; margin-bottom:0px">Our 
direct-connected&nbsp;site,&nbsp;which is using a 10-way split right now, also 
shows a return to calmness in the latencies:</p>
<p style="margin-top:0px; margin-bottom:0px"><img size="25968" 
style="width:736px; max-width:100%" data-outlook-trace="F:2|T:2" 
src="cid:16a02dbca53f456b1e51"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Prior to the recent latency jump, 
I did not use split requests and the reception had been stellar for quite some 
time.&nbsp; It's my suspicion that this is a networking congestion issue 
somewhere close to the source since
 it seems to affect all downstream sites.&nbsp; For that reason, I don't think 
solving this problem should necessarily involve upgrading your server software, 
but rather identifying what's jamming up the network near D.C., and testing 
this by switching to Boulder
 was an excellent idea.&nbsp; I will now try switching our system to a two-way 
split to see if this performance holds up with fewer pipes.&nbsp;&nbsp;<span 
style="font-size:12pt">Thanks for your help and I'll let you know what I find 
out.</span></p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
&nbsp;Art</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif,EmojiFont,&quot;Apple Color 
Emoji&quot;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Segoe UI 
Symbol&quot;,&quot;Android Emoji&quot;,EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Arthur A. Person</span><br 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Assistant Research Professor, System 
Administrator</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Penn State Department of Meteorology and 
Atmospheric Science</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">email:</span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><span><a 
href="mailto:aap1@xxxxxxx"; target="_blank">aap1@xxxxxxx</a></span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">,
 phone:</span><span style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><a 
href="callto:814-863-1563" target="_blank" 
style="color:rgb(0,61,121)">814-863-1563</a></span><span></span><br>
</p>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b>
<a href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a> &lt;<a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a>&gt; on behalf of Gilbert 
Sebenste &lt;<a href="mailto:gilbert@xxxxxxxxxxxxxxxx"; 
target="_blank">gilbert@xxxxxxxxxxxxxxxx</a>&gt;<br>
<b>Sent:</b> Wednesday, April 3, 2019 4:07 PM<br>
<b>To:</b> Anne Myckow - NOAA Affiliate<br>
<b>Cc:</b> <a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>; _NCEP.List.pmb-dataflow;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT 
feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
<div>
<div dir="ltr">Hello Anne,<br>
<br>
I'll jump in here as well. Consider the CONDUIT delays at UNIDATA:<br>
<br>
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bconduit.unidata.ucar.edu&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262109744&amp;sdata=gTeKW1u6tq4bdO8jRpmrxYQYG%2BDwwNUZBCseIInuftQ%3D&amp;reserved=0";
 
target="_blank">http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu</a>&nbsp;<br>
<br>
And now, Wisconsin:&nbsp;<br>
<br>
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bidd.aos.wisc.edu&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262109744&amp;sdata=7u4RkBCyKcWO16hQ9yieCa%2Bi2y%2BwhOWHtKPtI679dCI%3D&amp;reserved=0";
 
target="_blank">http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu</a>&nbsp;<br>
<br>
And finally, the University of Washington:<br>
<br>
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bfreshair1.atmos.washington.edu&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262119753&amp;sdata=31845ZclhIciL%2F3nuo682EDU%2FEiJXjXogqUqOUzNji4%3D&amp;reserved=0";
 
target="_blank">http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+freshair1.atmos.washington.edu</a>&nbsp;&nbsp;<br>
<br>
All three of whom have direct feeds from you. Flipping over to Boulder 
definitely caused a major improvement. There was still a brief spike in delay, 
but much shorter and minimal
<div>compared to what it was.<br>
<br>
Gilbert</div>
</div>
<br>
<div>
<div dir="ltr">On Wed, Apr 3, 2019 at 10:03 AM Anne Myckow - NOAA Affiliate 
&lt;<a href="mailto:anne.myckow@xxxxxxxx"; 
target="_blank">anne.myckow@xxxxxxxx</a>&gt; wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid 
rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hi Pete,
<div><br>
</div>
<div>As of yesterday we failed almost all of our applications to our site in 
Boulder (meaning away from CONDUIT). Have you noticed an improvement in your 
speeds since yesterday afternoon? If so this will give us a clue that maybe 
there's something interfering
 on our side that isn't specifically CONDUIT, but another app that might be 
causing congestion. (And if it's the same then that's a clue in the other 
direction.)</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Anne</div>
</div>
<br>
<div>
<div dir="ltr">On Mon, Apr 1, 2019 at 3:24 PM Pete Pokrandt &lt;<a 
href="mailto:poker@xxxxxxxxxxxx"; target="_blank">poker@xxxxxxxxxxxx</a>&gt; 
wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid 
rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
The lag here at UW-Madison was up to 1200 seconds today, and that's with a 
10-way split feed. Whatever is causing the issue has definitely not been 
resolved, and historically is worse during the work week than on the weekends. 
If that helps at all.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262129771&amp;sdata=bVTgXqm4bybswNIYttm3NeKaN5prjmUDwOpFXphUMCw%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b> Anne Myckow - NOAA Affiliate &lt;<a 
href="mailto:anne.myckow@xxxxxxxx"; 
target="_blank">anne.myckow@xxxxxxxx</a>&gt;<br>
<b>Sent:</b> Thursday, March 28, 2019 4:28 PM<br>
<b>To:</b> Person, Arthur A.<br>
<b>Cc:</b> Carissa Klemmer - NOAA Federal; Pete Pokrandt; 
_NCEP.List.pmb-dataflow;
<a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - 
started a week or so ago</font>
<div>&nbsp;</div>
</div>
<div>
<div>
<div dir="auto">Hello Art,</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">We will not be upgrading to version 6.13 on these systems as 
they are not robust enough to support the local logging inherent in the new 
version.</div>
<div dir="auto"><br>
</div>
<div dir="auto">I will check in with my team on if there are any further 
actions we can take to try and troubleshoot this issue, but I fear we may be at 
the limit of our ability to make this better.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Iâll let you know tomorrow where we stand. Thanks.</div>
<div dir="auto">Anne</div>
<div><br>
<div>
<div dir="ltr">On Mon, Mar 25, 2019 at 3:00 PM Person, Arthur A. &lt;<a 
href="mailto:aap1@xxxxxxx"; target="_blank">aap1@xxxxxxx</a>&gt; wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid 
rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px">Carissa,</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Can you report any status on this 
inquiry?</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Thanks...&nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; Art</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif,EmojiFont,&quot;Apple Color 
Emoji&quot;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Segoe UI 
Symbol&quot;,&quot;Android Emoji&quot;,EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Arthur A. Person</span><br 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Assistant Research Professor, System 
Administrator</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Penn State Department of Meteorology and 
Atmospheric Science</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">email:</span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><span><a 
href="mailto:aap1@xxxxxxx"; target="_blank">aap1@xxxxxxx</a></span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">,
 phone:</span><span style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><a 
href="callto:814-863-1563" target="_blank" 
style="color:rgb(0,61,121)">814-863-1563</a></span><span></span><br>
</p>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b> Carissa Klemmer - NOAA Federal &lt;<a 
href="mailto:carissa.l.klemmer@xxxxxxxx"; 
target="_blank">carissa.l.klemmer@xxxxxxxx</a>&gt;<br>
<b>Sent:</b> Tuesday, March 12, 2019 8:30 AM<br>
<b>To:</b> Pete Pokrandt<br>
<b>Cc:</b> Person, Arthur A.; <a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">
conduit@xxxxxxxxxxxxxxxx</a>; <a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">
support-conduit@xxxxxxxxxxxxxxxx</a>; _NCEP.List.pmb-dataflow<br>
<b>Subject:</b> Re: Large lags on CONDUIT feed - started a week or so ago</font>
<div>&nbsp;</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div>Hi Everyone
<div><br>
</div>
<div>Iâve added the Dataflow team email to the thread. I havenât heard that any 
changes were made or that any issues were found. But the team can look today 
and see if we have any signifiers of overall slowness with anything.&nbsp;</div>
<div><br>
</div>
<div>Dataflow, try taking a look at the new Citrix or VM troubleshooting tools 
if there are any abnormal signatures that may explain this.&nbsp;<br>
<br>
On Monday, March 11, 2019, Pete Pokrandt &lt;<a 
href="mailto:poker@xxxxxxxxxxxx"; target="_blank">poker@xxxxxxxxxxxx</a>&gt; 
wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid 
rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Art,</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
I don't know if NCEP ever figured anything out, but I've been able to keep my 
latencies reasonable (300-600s max, mostly during the 12 UTC model suite) by 
splitting my CONDUIT request 10 ways, instead of the 5 that I had been doing, 
or in a single request.
 Maybe give that a try and see if it helps at all.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262129771&amp;sdata=bVTgXqm4bybswNIYttm3NeKaN5prjmUDwOpFXphUMCw%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b> Person, Arthur A. &lt;<a 
href="mailto:aap1@xxxxxxx"; target="_blank">aap1@xxxxxxx</a>&gt;<br>
<b>Sent:</b> Monday, March 11, 2019 3:45 PM<br>
<b>To:</b> Holly Uhlenhake - NOAA Federal; Pete Pokrandt<br>
<b>Cc:</b> <a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] Large lags on CONDUIT feed - started a week or so 
ago</font>
<div>&nbsp;</div>
</div>
<div dir="ltr">
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px">Holly,</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Was there any resolution to this 
on the NCEP end?&nbsp; I'm still seeing terrible delays (1000-4000 seconds) 
receiving data from
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fconduit.ncep.noaa.gov&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262139772&amp;sdata=g9nb3WC8rR%2F65E%2FwZg%2BAwl7JDuNM6OV6ty5YNNg0YQ8%3D&amp;reserved=0";
 target="_blank">
conduit.ncep.noaa.gov</a>.&nbsp; It would be helpful to know if things are 
resolved at NCEP's end so I know whether to look further down the line.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Thanks...&nbsp; &nbsp; &nbsp; 
&nbsp; &nbsp; &nbsp;Art</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
font-family:Calibri,Helvetica,sans-serif,EmojiFont,&quot;Apple Color 
Emoji&quot;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Segoe UI 
Symbol&quot;,&quot;Android Emoji&quot;,EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Arthur A. Person</span><br 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Assistant Research Professor, System 
Administrator</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">Penn State Department of Meteorology and 
Atmospheric Science</span><br style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">
<span style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">email:</span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><span><a 
href="mailto:aap1@xxxxxxx"; target="_blank">aap1@xxxxxxx</a></span></span><span 
style="font-family:arial,helvetica,sans-serif; font-size:16px; 
background-color:rgb(253,253,253)">,
 phone:</span><span style="font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)">&nbsp;&nbsp;</span><span 
style="color:rgb(0,61,121); font-family:arial,helvetica,sans-serif; 
font-size:16px; background-color:rgb(253,253,253)"><a 
href="callto:814-863-1563" target="_blank" 
style="color:rgb(0,61,121)">814-863-1563</a></span><span></span><br>
</p>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b>
<a href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a> &lt;<a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a>&gt; on behalf of Holly 
Uhlenhake - NOAA Federal &lt;<a href="mailto:holly.uhlenhake@xxxxxxxx"; 
target="_blank">holly.uhlenhake@xxxxxxxx</a>&gt;<br>
<b>Sent:</b> Thursday, February 21, 2019 12:05 PM<br>
<b>To:</b> Pete Pokrandt<br>
<b>Cc:</b> <a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] Large lags on CONDUIT feed - started a week or so 
ago</font>
<div>&nbsp;</div>
</div>
<div>
<div dir="ltr">
<div>Hi Pete,</div>
<div><br>
</div>
<div>We'll take a look and see if we can figure out what might be going 
on.&nbsp; We haven't done anything to try and address this yet, but based on 
your analysis I'm suspicious that it might be tied to a resource constraint on 
the VM or the blade it resides on.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Holly Uhlenhake</div>
<div>Acting Dataflow Team Lead <br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Feb 21, 2019 at 11:32 AM Pete Pokrandt &lt;<a 
href="mailto:poker@xxxxxxxxxxxx"; target="_blank">poker@xxxxxxxxxxxx</a>&gt; 
wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid 
rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Just FYI, data is flowing, but the large lags continue.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bidd.aos.wisc.edu&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262139772&amp;sdata=llapIFZjpMj%2F8T3WDF5rL8yqOJs6lCOg6z39xzhfZAQ%3D&amp;reserved=0";
 
target="_blank">http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu</a><br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bconduit.unidata.ucar.edu&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262149772&amp;sdata=cDjmr2BO09PdgUDjkTaIMAVzNaBZG8zoI2DlLM34ffg%3D&amp;reserved=0";
 
target="_blank">http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu</a><br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262159781&amp;sdata=hzjBmCU6jS8yHYLmO%2FItc68jCbz790Z6x7960AWFJcI%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b>
<a href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a> &lt;<a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a>&gt; on behalf of Pete 
Pokrandt &lt;<a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a>&gt;<br>
<b>Sent:</b> Wednesday, February 20, 2019 12:07 PM<br>
<b>To:</b> Carissa Klemmer - NOAA Federal<br>
<b>Cc:</b> <a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] Large lags on CONDUIT feed - started a week or so 
ago</font>
<div>&nbsp;</div>
</div>
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Data is flowing again - picked up somewhere in the GEFS. Maybe CONDUIT server 
was restarted, or ldm on it? Lags are large (3000s+) but dropping slowly<br>
<br>
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262159781&amp;sdata=hzjBmCU6jS8yHYLmO%2FItc68jCbz790Z6x7960AWFJcI%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b>
<a href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a> &lt;<a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a>&gt; on behalf of Pete 
Pokrandt &lt;<a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a>&gt;<br>
<b>Sent:</b> Wednesday, February 20, 2019 11:56 AM<br>
<b>To:</b> Carissa Klemmer - NOAA Federal<br>
<b>Cc:</b> <a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> Re: [conduit] Large lags on CONDUIT feed - started a week or so 
ago</font>
<div>&nbsp;</div>
</div>
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Just a quick follow-up - we started falling far enough behind (3600+ sec) that 
we are losing data. We got short files starting at 174h into the GFS run, and 
only got (incomplete) data through 207h.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
We have now not received any data on CONDUIT since 11:27 AM CST (1727 UTC) 
today (Wed Feb 20)</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262169786&amp;sdata=M8%2BWo8v1Gdkfo5UseX%2FAZ3Q2T07nEJy3DFqDJvaqT8Q%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
<div>
<div></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" 
style="font-size:11pt"><b>From:</b>
<a href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a> &lt;<a 
href="mailto:conduit-bounces@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit-bounces@xxxxxxxxxxxxxxxx</a>&gt; on behalf of Pete 
Pokrandt &lt;<a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a>&gt;<br>
<b>Sent:</b> Wednesday, February 20, 2019 11:28 AM<br>
<b>To:</b> Carissa Klemmer - NOAA Federal<br>
<b>Cc:</b> <a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a>;
<a href="mailto:support-conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">support-conduit@xxxxxxxxxxxxxxxx</a><br>
<b>Subject:</b> [conduit] Large lags on CONDUIT feed - started a week or so 
ago</font>
<div>&nbsp;</div>
</div>
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Carissa,</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
We have been feeding CONDUIT using a 5 way split feed direct from <a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fconduit.ncep.noaa.gov&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262169786&amp;sdata=pwQ8RllTCU%2BOL0uliFgC9G0V%2F6WK%2FhLaOn%2B2QL%2F1pcE%3D&amp;reserved=0";
 target="_blank">
conduit.ncep.noaa.gov</a>, and it had been really good for some time, lags 
30-60 seconds or less.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
However, the past week or so, we've been seeing some very large lags during 
each 6 hour model suite - Unidata is also seeing these - they are also feeding 
direct from
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fconduit.ncep.noaa.gov&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262179795&amp;sdata=1xxCP%2FafYbMXdcLNR%2FIuy9Y0tBDO%2FKWbBYsAPjLov6w%3D&amp;reserved=0";
 target="_blank">
conduit.ncep.noaa.gov</a>.</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bidd.aos.wisc.edu&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262189808&amp;sdata=qJfe%2FdGVmh8p3E%2BaMelwyjvLsxXOuodmdjYhfGM%2BbH0%3D&amp;reserved=0";
 
target="_blank">http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu</a><br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bconduit.unidata.ucar.edu&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262189808&amp;sdata=JFmzqxz80IM%2FSiNV1zl6QeWwcIH8miS8zTyFarxqS8M%3D&amp;reserved=0";
 
target="_blank">http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu</a><br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Any idea what's going on, or how we can find out?&nbsp;<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Thanks!</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
Pete</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:12pt; 
color:rgb(0,0,0)">
<br>
</div>
<div>
<div dir="ltr" style="font-size:12pt; color:rgb(0,0,0); 
background-color:rgb(255,255,255); 
font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"><a title="Ctrl+Click or tap to 
follow the link" 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262199813&amp;sdata=6U6qIL0jgRoiQ742NlkHr344BFJqthlSVRGoKuyovQU%3D&amp;reserved=0";
 target="_blank"></a>--<br>
Pete Pokrandt - Systems Programmer<br>
UW-Madison Dept of Atmospheric and Oceanic Sciences<br>
608-262-3086&nbsp; - <a href="mailto:poker@xxxxxxxxxxxx"; 
target="_blank">poker@xxxxxxxxxxxx</a><br>
</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
NOTE: All exchanges posted to Unidata maintained email lists are<br>
recorded in the Unidata inquiry tracking system and made publicly<br>
available through the web.&nbsp; Users who post to any of the lists we<br>
maintain are reminded to remove any personal information that they<br>
do not want to be made public.<br>
<br>
<br>
conduit mailing list<br>
<a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a><br>
For list information or to unsubscribe, visit: <a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262199813&amp;sdata=cTlTde6vBUC%2Bc%2BPnOlfuJsSbdma9urfLTGB5CXjxtxc%3D&amp;reserved=0";
 rel="noreferrer" target="_blank">
http://www.unidata.ucar.edu/mailing_lists/</a> </blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br>
-- <br>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><font size="2"><span style="color:rgb(102,102,102); 
font-family:georgia,serif">Carissa Klemmer
</span></font><br style="color:rgb(102,102,102); font-family:georgia,serif">
<font size="2"><span style="color:rgb(102,102,102); 
font-family:georgia,serif">NCEP Central Operations
</span></font><br style="color:rgb(102,102,102); font-family:georgia,serif">
<font size="2"><span style="color:rgb(102,102,102); 
font-family:georgia,serif">IDSB Branch Chief</span></font></div>
<div><font size="2"><span style="color:rgb(102,102,102); 
font-family:georgia,serif"></span></font><font size="2"><span 
style="color:rgb(102,102,102); 
font-family:georgia,serif">301-683-3835</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Ncep.list.pmb-dataflow mailing list<br>
<a href="mailto:Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx"; 
target="_blank">Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx</a><br>
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.lstsrv.ncep.noaa.gov%2Fmailman%2Flistinfo%2Fncep.list.pmb-dataflow&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262209822&amp;sdata=mbUF1EKDyPVMZlUnjn86XQ55KuTqbJIZYaDKwOFuy60%3D&amp;reserved=0";
 rel="noreferrer" 
target="_blank">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow</a><br>
</blockquote>
</div>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div>Anne Myckow<br>
</div>
Lead Dataflow Analyst<br>
</div>
NOAA/NCEP/NCO<br>
301-683-3825<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div>Anne Myckow<br>
</div>
Lead Dataflow Analyst<br>
</div>
NOAA/NCEP/NCO<br>
301-683-3825<br>
</div>
</div>
</div>
</div>
_______________________________________________<br>
NOTE: All exchanges posted to Unidata maintained email lists are<br>
recorded in the Unidata inquiry tracking system and made publicly<br>
available through the web.&nbsp; Users who post to any of the lists we<br>
maintain are reminded to remove any personal information that they<br>
do not want to be made public.<br>
<br>
<br>
conduit mailing list<br>
<a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a><br>
For list information or to unsubscribe, visit: <a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262209822&amp;sdata=wPUwExB77bt53ZiYByOJ40e%2FQAP1ByJYo3bDTbubgZI%3D&amp;reserved=0";
 rel="noreferrer" target="_blank">
http://www.unidata.ucar.edu/mailing_lists/</a> </blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div>----</div>
<div>&nbsp;</div>
<div>Gilbert Sebenste</div>
<div>Consulting Meteorologist</div>
<div>AllisonHouse, LLC</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Ncep.list.pmb-dataflow mailing list<br>
<a href="mailto:Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx"; 
target="_blank">Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx</a><br>
<a 
href="https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.lstsrv.ncep.noaa.gov%2Fmailman%2Flistinfo%2Fncep.list.pmb-dataflow&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262219827&amp;sdata=wQavNwdnZST2BGlnWXTo0cM7yCZKfC89ELJVz6I9wTA%3D&amp;reserved=0";
 rel="noreferrer" 
target="_blank">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow</a><br>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div>Derek Van Pelt<br>
</div>
DataFlow Analyst<br>
</div>
NOAA/NCEP/NCO<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br>
-- <br>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><font size="2"><span style="color:rgb(102,102,102); 
font-family:georgia,serif">Carissa Klemmer
</span></font><br style="color:rgb(102,102,102); font-family:georgia,serif">
<font size="2"><span style="color:rgb(102,102,102); 
font-family:georgia,serif">NCEP Central Operations
</span></font><br style="color:rgb(102,102,102); font-family:georgia,serif">
<font size="2"><span style="color:rgb(102,102,102); 
font-family:georgia,serif">IDSB Branch Chief</span></font></div>
<div><font size="2"><span style="color:rgb(102,102,102); 
font-family:georgia,serif"></span></font><font size="2"><span 
style="color:rgb(102,102,102); 
font-family:georgia,serif">301-683-3835</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
NOTE: All exchanges posted to Unidata maintained email lists are<br>
recorded in the Unidata inquiry tracking system and made publicly<br>
available through the web.&nbsp; Users who post to any of the lists we<br>
maintain are reminded to remove any personal information that they<br>
do not want to be made public.<br>
<br>
<br>
conduit mailing list<br>
<a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a><br>
For list information or to unsubscribe, visit: <a 
href="https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&amp;data=02%7C01%7Caap1%40psu.edu%7C03f4b8f758bb49bc0bb408d6bd00fc16%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636904209262229827&amp;sdata=J4CywBaeCryhk3m%2F912pEFCmFpsdz%2BUVxwcMXguJXik%3D&amp;reserved=0";
 rel="noreferrer" target="_blank">
http://www.unidata.ucar.edu/mailing_lists/</a> </blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_m_341510715877528186gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div>Derek Van Pelt<br>
</div>
DataFlow Analyst<br>
</div>
NOAA/NCEP/NCO<br>
</div>
</div>
</div>
</div>
</div>
</div>
-- <br>
<div dir="ltr" 
class="x_x_gmail-m_4641072781522989288gmail-m_3586078264856567528x_gmail_signature">
Misspelled straight from Derek's phone.</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
NOTE: All exchanges posted to Unidata maintained email lists are<br>
recorded in the Unidata inquiry tracking system and made publicly<br>
available through the web.&nbsp; Users who post to any of the lists we<br>
maintain are reminded to remove any personal information that they<br>
do not want to be made public.<br>
<br>
<br>
conduit mailing list<br>
<a href="mailto:conduit@xxxxxxxxxxxxxxxx"; 
target="_blank">conduit@xxxxxxxxxxxxxxxx</a><br>
For list information or to unsubscribe, visit: <a 
href="http://www.unidata.ucar.edu/mailing_lists/"; rel="noreferrer" 
target="_blank">
http://www.unidata.ucar.edu/mailing_lists/</a> </blockquote>
</div>
_______________________________________________<br>
Ncep.list.pmb-dataflow mailing list<br>
<a href="mailto:Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx"; 
target="_blank">Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx</a><br>
<a 
href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow"; 
rel="noreferrer" 
target="_blank">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow</a><br>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="x_x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><span style="color:rgb(0,0,0)"><span 
style="color:rgb(68,68,68)">Dustin Sheffler</span></span><br>
<div><span style="color:rgb(102,102,102)"><span 
style="color:rgb(68,68,68)">NCEP Central Operations - Dataflow</span></span><br>
<span><a href="tel:%28301%29%20683-1400" value="+13016831400" 
target="_blank"><span><span></span></span></a></span><span 
style="color:rgb(102,102,102)"><span style="color:rgb(68,68,68)"><span 
dir="ltr">5830 University Research Court</span>, Rm 1030<br>
<span dir="ltr">College Park, Maryland</span> 20740<br>
</span></span>
<div style="text-align:left"><span style="color:rgb(102,102,102)"><span 
style="color:rgb(68,68,68)">Office:</span>
</span><a href="tel:%28301%29%20683-1400" value="+13016831400" 
target="_blank">(301) 683-3827</a><br>
<div style="text-align:left"><a href="tel:%28301%29%20683-1400" 
value="+13016831400" 
target="_blank"><span><span></span></span></a><span><span><span><span><span><span><span><span></span></span></span></span></span></span></span></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>


</div></blockquote><blockquote type="cite"><div 
dir="ltr"><span>_______________________________________________</span><br><span>NOTE:
 All exchanges posted to Unidata maintained email lists 
are</span><br><span>recorded in the Unidata inquiry tracking system and made 
publicly</span><br><span>available through the web. &nbsp;Users who post to any 
of the lists we</span><br><span>maintain are reminded to remove any personal 
information that they</span><br><span>do not want to be made 
public.</span><br><span></span><br><span></span><br><span>conduit mailing 
list</span><br><span><a 
href="mailto:conduit@xxxxxxxxxxxxxxxx";>conduit@xxxxxxxxxxxxxxxx</a></span><br><span>For
 list information or to unsubscribe, visit: <a 
href="http://www.unidata.ucar.edu/mailing_lists/";>http://www.unidata.ucar.edu/mailing_lists/</a>
 </span></div></blockquote></body></html>