Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - started a week or so ago

  • To: "Person, Arthur A." <aap1@xxxxxxx>
  • Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - started a week or so ago
  • From: Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
  • Date: Thu, 18 Apr 2019 17:47:02 -0500
Internet2 doesn't packet shape. NCEP, or your University, can do packet 
shaping. 

When I worked at NIU, I had the IT department set up a rule so that my servers 
could not be packet shaped inbound or outbound. When they first set it up, I 
couldn't get data in some peak periods.

Gilbert

> On Apr 18, 2019, at 2:20 PM, Person, Arthur A. <aap1@xxxxxxx> wrote:
> 
> All --
> 
> I switched our test system, iddrs2a, feeding from conduit.ncep.noaa.gov from 
> a 2-way split to a 20-way split yesterday, and the results are dramatic:
> 
> <pastedImage.png>
> Although conduit feed performance at other sites improved a little last night 
> with the MRMS feed failure, it doesn't explain this improvement entirely.  
> This leads me to ponder the causes of such an improvement:
> 
> 1) The network path does not appear to be bandwidth constrained, otherwise 
> there would be no improvement no matter how many pipes were used;
> 
> 2) The problem, therefore, would appear to be packet oriented, either with 
> path packet saturation, or packet shaping.
> 
> I'm not a networking expert, so maybe I'm missing another possibility here, 
> but I'm curious whether packet shaping could account for some of the 
> throughput issues.  I've also been having trouble getting timely delivery of 
> our Unidata IDD satellite feed, and discovered that switching that to a 
> 10-way split feed (from a 2-way split) has reduced the latencies from 
> 2000-3000 seconds down to less than 300 seconds.  Interestingly, the peak 
> satellite feed latencies (see below) occur at the same time as the peak 
> conduit latencies, but this path is unrelated to NCEP (as far as I know).  Is 
> it possible that Internet 2 could be packet-shaping their traffic and that 
> this could be part of the cause for the packet latencies we're seeing?
> 
>                              Art
> 
> <pastedImage.png>
> 
> 
> Arthur A. Person
> Assistant Research Professor, System Administrator
> Penn State Department of Meteorology and Atmospheric Science
> email:  aap1@xxxxxxx, phone:  814-863-1563
> 
> 
>  
> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> on 
> behalf of Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
> Sent: Thursday, April 18, 2019 2:29 AM
> To: Pete Pokrandt
> Cc: Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; Mike 
> Zuranski; Derek VanPelt - NOAA Affiliate; Dustin Sheffler - NOAA Federal; 
> support-conduit@xxxxxxxxxxxxxxxx
> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - 
> started a week or so ago
>  
> FYI: all evening and into the overnight, MRMS data has been missing, QC BR 
> has been town for the last 40 minutes, but smaller products are coming 
> through somewhat more reliably as of 6Z. CONDUIT was still substantially 
> delayed around 4Z with the GFS.
> 
> Gilbert
> 
> On Apr 16, 2019, at 5:43 PM, Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
> 
>> Here's a few traceroutes from just now - from idd-agg.aos.wisc.edu to 
>> conduit.ncep.noaa.gov. The lags are up and running around 600-800 seconds 
>> right now. I'm not including all of the * * * lines from after 144.90.76.65 
>> which is presumably behind a firewall.
>> 
>> 
>> 2209 UTC Tuesday Apr 16
>> 
>> traceroute -p 388 conduit.ncep.noaa.gov
>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte 
>> packets
>>  1  vlan-510-cssc-gw.net.wisc.edu (144.92.130.1)  0.906 ms  0.701 ms  0.981 
>> ms
>>  2  128.104.4.129 (128.104.4.129)  1.700 ms  1.737 ms  1.772 ms
>>  3  rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)  
>> 1.740 ms  3.343 ms  3.336 ms
>>  4  rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu (146.151.166.122)  
>> 2.043 ms  2.034 ms  1.796 ms
>>  5  144.92.254.229 (144.92.254.229)  11.530 ms  11.472 ms  11.535 ms
>>  6  et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60)  22.813 ms  
>> 22.899 ms  22.886 ms
>>  7  et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2)  24.248 ms  
>> 24.195 ms  24.172 ms
>>  8  nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189)  24.244 ms  24.196 
>> ms  24.183 ms
>>  9  ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4)  24.937 ms  24.884 ms  
>> 24.878 ms
>> 10  140.208.63.30 (140.208.63.30)  134.030 ms  126.195 ms  126.305 ms
>> 11  140.90.76.65 (140.90.76.65)  106.810 ms  104.553 ms  104.603 ms
>> 
>> 2230 UTC Tuesday Apr 16
>> 
>> traceroute -p 388 conduit.ncep.noaa.gov
>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte 
>> packets
>>  1  vlan-510-cssc-gw.net.wisc.edu (144.92.130.1)  1.391 ms  1.154 ms  5.902 
>> ms
>>  2  128.104.4.129 (128.104.4.129)  6.917 ms  6.895 ms  2.004 ms
>>  3  rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)  
>> 3.158 ms  3.293 ms  3.251 ms
>>  4  rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu (146.151.166.122)  
>> 6.185 ms  2.278 ms  2.425 ms
>>  5  144.92.254.229 (144.92.254.229)  6.909 ms  13.255 ms  6.863 ms
>>  6  et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60)  23.328 ms  
>> 23.244 ms  28.845 ms
>>  7  et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2)  30.308 ms  
>> 24.575 ms  24.536 ms
>>  8  nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189)  29.594 ms  24.624 
>> ms  24.618 ms
>>  9  ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4)  24.581 ms  30.164 ms  
>> 24.627 ms
>> 10  140.208.63.30 (140.208.63.30)  25.677 ms  25.767 ms  29.543 ms
>> 11  140.90.76.65 (140.90.76.65)  105.812 ms  105.345 ms  108.857
>> 
>> 2232 UTC Tuesday Apr 16
>> 
>> traceroute -p 388 conduit.ncep.noaa.gov
>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte 
>> packets
>>  1  vlan-510-cssc-gw.net.wisc.edu (144.92.130.1)  1.266 ms  1.070 ms  1.226 
>> ms
>>  2  128.104.4.129 (128.104.4.129)  1.915 ms  2.652 ms  2.775 ms
>>  3  rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)  
>> 2.353 ms  2.129 ms  2.314 ms
>>  4  rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu (146.151.166.122)  
>> 2.114 ms  2.111 ms  2.163 ms
>>  5  144.92.254.229 (144.92.254.229)  6.891 ms  6.838 ms  6.840 ms
>>  6  et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60)  23.336 ms  
>> 23.283 ms  23.364 ms
>>  7  et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2)  24.493 ms  
>> 24.136 ms  24.152 ms
>>  8  nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189)  24.161 ms  24.173 
>> ms  24.176 ms
>>  9  ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4)  24.165 ms  24.331 ms  
>> 24.201 ms
>> 10  140.208.63.30 (140.208.63.30)  25.361 ms  25.427 ms  25.240 ms
>> 11  140.90.76.65 (140.90.76.65)  113.194 ms  115.553 ms  115.543 ms
>> 
>> 
>> 2234 UTC Tuesday Apr 16
>> 
>> traceroute -p 388 conduit.ncep.noaa.gov
>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte 
>> packets
>>  1  vlan-510-cssc-gw.net.wisc.edu (144.92.130.1)  0.901 ms  0.663 ms  0.826 
>> ms
>>  2  128.104.4.129 (128.104.4.129)  1.645 ms  1.948 ms  1.729 ms
>>  3  rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)  
>> 1.804 ms  1.788 ms  1.849 ms
>>  4  rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu (146.151.166.122)  
>> 2.011 ms  2.004 ms  1.982 ms
>>  5  144.92.254.229 (144.92.254.229)  6.241 ms  6.240 ms  6.220 ms
>>  6  et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60)  23.042 ms  
>> 23.072 ms  23.033 ms
>>  7  et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2)  24.094 ms  
>> 24.398 ms  24.370 ms
>>  8  nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189)  24.166 ms  24.166 
>> ms  24.108 ms
>>  9  ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4)  24.056 ms  24.306 ms  
>> 24.215 ms
>> 10  140.208.63.30 (140.208.63.30)  25.199 ms  25.284 ms  25.351 ms
>> 11  140.90.76.65 (140.90.76.65)  118.314 ms  118.707 ms  118.768 ms
>> 
>> 2236 UTC Tuesday Apr 16
>> 
>> traceroute -p 388 conduit.ncep.noaa.gov
>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte 
>> packets
>>  1  vlan-510-cssc-gw.net.wisc.edu (144.92.130.1)  0.918 ms  0.736 ms  0.864 
>> ms
>>  2  128.104.4.129 (128.104.4.129)  1.517 ms  1.630 ms  1.734 ms
>>  3  rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)  
>> 1.998 ms  3.437 ms  3.437 ms
>>  4  rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu (146.151.166.122)  
>> 1.899 ms  1.896 ms  1.867 ms
>>  5  144.92.254.229 (144.92.254.229)  6.384 ms  6.317 ms  6.314 ms
>>  6  et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60)  22.980 ms  
>> 23.167 ms  23.078 ms
>>  7  et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2)  24.181 ms  
>> 24.152 ms  24.121 ms
>>  8  nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189)  48.556 ms  47.824 
>> ms  47.799 ms
>>  9  ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4)  24.166 ms  24.154 ms  
>> 24.214 ms
>> 10  140.208.63.30 (140.208.63.30)  25.310 ms  25.268 ms  25.401 ms
>> 11  140.90.76.65 (140.90.76.65)  118.299 ms  123.763 ms  122.207 ms
>> 
>> 2242 UTC 
>> 
>> traceroute -p 388 conduit.ncep.noaa.gov
>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte 
>> packets
>>  1  vlan-510-cssc-gw.net.wisc.edu (144.92.130.1)  1.337 ms  1.106 ms  1.285 
>> ms
>>  2  128.104.4.129 (128.104.4.129)  6.039 ms  5.778 ms  1.813 ms
>>  3  rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)  
>> 2.275 ms  2.464 ms  2.517 ms
>>  4  rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu (146.151.166.122)  
>> 2.288 ms  6.978 ms  3.506 ms
>>  5  144.92.254.229 (144.92.254.229)  10.369 ms  6.626 ms  10.281 ms
>>  6  et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60)  23.513 ms  
>> 23.297 ms  23.295 ms
>>  7  et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2)  27.938 ms  
>> 24.589 ms  28.783 ms
>>  8  nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189)  28.796 ms  24.630 
>> ms  28.793 ms
>>  9  ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4)  24.576 ms  24.545 ms  
>> 24.587 ms
>> 10  140.208.63.30 (140.208.63.30)  85.763 ms  85.768 ms  83.623 ms
>> 11  140.90.76.65 (140.90.76.65)  131.912 ms  132.662 ms  132.340 ms
>> 
>> Pete
>> 
>> --
>> Pete Pokrandt - Systems Programmer
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>> 608-262-3086  - poker@xxxxxxxxxxxx
>> 
>>  
>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> on 
>> behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
>> Sent: Tuesday, April 16, 2019 3:04 PM
>> To: Gilbert Sebenste; Tyle, Kevin R
>> Cc: Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; Derek 
>> VanPelt - NOAA Affiliate; Mike Zuranski; Dustin Sheffler - NOAA Federal; 
>> support-conduit@xxxxxxxxxxxxxxxx
>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - 
>> started a week or so ago
>>  
>> At UW-Madison, we had incomplete 12 UTC GFS data starting with the 177h 
>> forecast. Lags exceeded 3600s.
>> 
>> 
>> 
>> Pete
>> 
>> 
>> --
>> Pete Pokrandt - Systems Programmer
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>> 608-262-3086  - poker@xxxxxxxxxxxx
>> 
>>  
>> From: Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
>> Sent: Tuesday, April 16, 2019 2:44 PM
>> To: Tyle, Kevin R
>> Cc: Pete Pokrandt; Dustin Sheffler - NOAA Federal; Mike Zuranski; Derek 
>> VanPelt - NOAA Affiliate; Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; 
>> _NCEP.List.pmb-dataflow; support-conduit@xxxxxxxxxxxxxxxx
>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - 
>> started a week or so ago
>>  
>> Yes, here at AllisonHouse too...we can feed from a number of sites, and all 
>> of them were dropping GFS, and delayed by an hour.
>> 
>> Gilbert
>> 
>> On Apr 16, 2019, at 2:39 PM, Tyle, Kevin R <ktyle@xxxxxxxxxx> wrote:
>> 
>>> For what it's worth, our 12Z GFS data ingest was quite bad today ... many 
>>> lost products beyond F168 (we feed from UWisc-MSN primary and PSU 
>>> secondary).
>>> 
>>> _____________________________________________
>>> Kevin Tyle, M.S.; Manager of Departmental Computing
>>> Dept. of Atmospheric & Environmental Sciences   
>>> University at Albany
>>> Earth Science 235, 1400 Washington Avenue                        
>>> Albany, NY 12222
>>> Email: ktyle@xxxxxxxxxx
>>> Phone: 518-442-4578                             
>>> _____________________________________________
>>>  
>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> 
>>> on behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
>>> Sent: Tuesday, April 16, 2019 12:00 PM
>>> To: Dustin Sheffler - NOAA Federal; Mike Zuranski
>>> Cc: Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; Derek VanPelt - NOAA 
>>> Affiliate; _NCEP.List.pmb-dataflow; support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed 
>>> - started a week or so ago
>>>  
>>> All,
>>> 
>>> Just keeping this in the foreground. 
>>> 
>>> CONDUIT lags continue to be very large compared to what they were previous 
>>> to whatever changed back in February. Prior to that, we rarely saw lags 
>>> more than ~300s. Now they are routinely 1500-2000s at UW-Madison and Penn 
>>> State, and  over 3000s at Unidata - they appear to be on the edge of losing 
>>> data. This does not bode well with all of the IDP applications failing back 
>>> over to CP today..
>>> 
>>> Can we send you some traceroutes and you back to us to maybe try to isolate 
>>> where in the network this is happening? It feels like congestion or a bad 
>>> route somewhere - the lags seem to be worse on weekdays than weekends if 
>>> that helps at all.
>>> 
>>> Here are the current CONDUIT lags to UW-Madison, Penn State and Unidata.
>>> 
>>> 
>>> <iddstats_CONDUIT_idd_aos_wisc_edu_ending_20190416_1600UTC.gif>
>>> 
>>> <iddstats_CONDUIT_idd_meteo_psu_edu_ending_20190416_1600UTC.gif>
>>> 
>>> <iddstats_CONDUIT_conduit_unidata_ucar_edu_ending_20190416_1600UTC.gif>
>>> 
>>> 
>>> 
>>> 
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> 
>>> on behalf of Dustin Sheffler - NOAA Federal <dustin.sheffler@xxxxxxxx>
>>> Sent: Tuesday, April 9, 2019 12:52 PM
>>> To: Mike Zuranski
>>> Cc: conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; Derek VanPelt - NOAA 
>>> Affiliate; support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed 
>>> - started a week or so ago
>>>  
>>> Hi Mike,
>>> 
>>> Thanks for the feedback on NOMADS. We recently found a slowness issue when 
>>> NOMADS is running out of our Boulder data center that is being worked on by 
>>> our teams now that NOMADS is live out of the College Park data center. It's 
>>> hard sometimes to quantify whether slowness issues that are only being 
>>> reported by a handful of users is a result of something wrong in our data 
>>> center, a bad network path between a customer (possibly just from a 
>>> particular region of the country) and our data center, a local issue on the 
>>> customers' end, or any other reason that might cause slowness. 
>>> 
>>> Conduit is only ever run from our College Park data center. It's slowness 
>>> is not tied into the Boulder NOMADS issue, but it does seem to be at least 
>>> a little bit tied to which of our data centers NOMADS is running out of. 
>>> When NOMADS is in Boulder along with the majority of our other NCEP 
>>> applications, the strain on the College Park data center is minimal and 
>>> Conduit appears to be running better as a result. When NOMADS runs in 
>>> College Park (as it has since late yesterday) there is more strain on the 
>>> data center and Conduit appears (based on provided user graphs) to run a 
>>> bit worse around peak model times as a result. These are just my 
>>> observations and we are still investigating what may have changed that 
>>> caused the Conduit latencies to appear in the first place so that we can 
>>> resolve this potential constraint. 
>>> 
>>> -Dustin
>>> 
>>> On Tue, Apr 9, 2019 at 4:28 PM Mike Zuranski <zuranski@xxxxxxxxxxxxxxx> 
>>> wrote:
>>> Hi everyone,
>>> 
>>> I've avoided jumping into this conversation since I don't deal much with 
>>> Conduit these days, but Derek just mentioned something that I do have some 
>>> applicable feedback on...
>>> 
>>> > Two items happened last night.  1. NOMADS was moved back to College 
>>> > Park...
>>> 
>>> We get nearly all of our model data via NOMADS.  When it switched to 
>>> Boulder last week we saw a significant drop in download speeds, down to a 
>>> couple hundred KB/s or slower.  Starting last night, we're back to speeds 
>>> on the order of MB/s or tens of MB/s.  Switching back to College Park seems 
>>> to confirm for me something about routing from Boulder was responsible.  
>>> But again this was all on NOMADS, not sure if it's related to happenings on 
>>> Conduit.
>>> 
>>> When I noticed this last week I sent an email to sdm@xxxxxxxx including a 
>>> traceroute taken at the time, let me know if you'd like me to find that and 
>>> pass it along here or someplace else.
>>> 
>>> -Mike
>>> 
>>> ======================
>>> Mike Zuranski
>>> Meteorology Support Analyst
>>> College of DuPage - Nexlab
>>> Weather.cod.edu
>>> ======================
>>> 
>>> 
>>> On Tue, Apr 9, 2019 at 10:51 AM Person, Arthur A. <aap1@xxxxxxx> wrote:
>>> Derek,
>>> 
>>> Do we know what change might have been made around February 10th when the 
>>> CONDUIT problems first started happening?  Prior to that time, the CONDUIT 
>>> feed had been very crisp for a long period of time.
>>> 
>>> Thanks...            Art
>>> 
>>> 
>>> Arthur A. Person
>>> Assistant Research Professor, System Administrator
>>> Penn State Department of Meteorology and Atmospheric Science
>>> email:  aap1@xxxxxxx, phone:  814-863-1563
>>> 
>>> 
>>>  
>>> From: Derek VanPelt - NOAA Affiliate <derek.vanpelt@xxxxxxxx>
>>> Sent: Tuesday, April 9, 2019 11:34 AM
>>> To: Holly Uhlenhake - NOAA Federal
>>> Cc: Carissa Klemmer - NOAA Federal; Person, Arthur A.; Pete Pokrandt; 
>>> _NCEP.List.pmb-dataflow; conduit@xxxxxxxxxxxxxxxx; 
>>> support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or so ago
>>>  
>>> Hi all,
>>> 
>>> Two items happened last night.
>>> 
>>> 1.   NOMADS was moved back to College Park, which means there was a lot 
>>> more traffic going out which will have effect on the Conduit latencies.  We 
>>> do not have a full load from the COllege Park Servers as many of the other 
>>> applications are still running from Boulder, but NOMADS will certainly 
>>> increase overall load.
>>> 
>>> 2.   As Holly said, there were further issues delaying and changing the 
>>> timing of the model output yesterday afternoon/evening.  I will be watching 
>>> from our end, and monitoring the Unidata 48 hour graph (thank you for the 
>>> link) throughout the day, 
>>> 
>>> Please let us know if you have questions or more information to help us 
>>> analyse what you are seeing.  
>>> 
>>> Thank you,
>>> 
>>> Derek
>>> 
>>> 
>>> On Tue, Apr 9, 2019 at 6:50 AM Holly Uhlenhake - NOAA Federal 
>>> <holly.uhlenhake@xxxxxxxx> wrote:
>>> Hi Pete,
>>> 
>>> We also had an issue on the supercomputer yesterday where several models 
>>> going to conduit would have been stacked on top of each other instead of 
>>> coming out in a more spread out fashion.  It's not inconceivable that 
>>> conduit could have backed up working through the abnormally large glut of 
>>> grib messages.    Are things better this morning at all?
>>> 
>>> Thanks,
>>> Holly
>>> 
>>> On Tue, Apr 9, 2019 at 12:37 AM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>> Something changed starting with today's 18 UTC model cycle, and our lags 
>>> shot up to over 3600 seconds, where we started losing data. They are 
>>> growing again now with the 00 UTC cycle as well. PSU and Unidata CONDUIT 
>>> stats show similar abnormally large lags.
>>> 
>>> FYI.
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: Person, Arthur A. <aap1@xxxxxxx>
>>> Sent: Friday, April 5, 2019 2:10 PM
>>> To: Carissa Klemmer - NOAA Federal
>>> Cc: Pete Pokrandt; Derek VanPelt - NOAA Affiliate; Gilbert Sebenste; 
>>> conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; 
>>> support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: Large lags on CONDUIT feed - started a week or so ago
>>>  
>>> Carissa,
>>> 
>>> The Boulder connection is definitely performing very well for CONDUIT.  
>>> Although there have been a couple of little blips (~ 120 seconds) since 
>>> yesterday, overall the performance is superb.  I don't think it's quite as 
>>> clean as prior to the ~February 10th date when the D.C. connection went 
>>> bad, but it's still excellent performance.  Here's our graph now with a 
>>> single connection (no splits):
>>> <pastedImage.png>
>>> My next question is:  Will CONDUIT stay pointing at Boulder until D.C. is 
>>> fixed, or might you be required to switch back to D.C. at some point before 
>>> that?
>>> 
>>> Thanks...               Art
>>> 
>>> Arthur A. Person
>>> Assistant Research Professor, System Administrator
>>> Penn State Department of Meteorology and Atmospheric Science
>>> email:  aap1@xxxxxxx, phone:  814-863-1563
>>> 
>>> 
>>>  
>>> From: Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>
>>> Sent: Thursday, April 4, 2019 6:22 PM
>>> To: Person, Arthur A.
>>> Cc: Pete Pokrandt; Derek VanPelt - NOAA Affiliate; Gilbert Sebenste; 
>>> conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; 
>>> support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: Large lags on CONDUIT feed - started a week or so ago
>>>  
>>> Catching up here.
>>> 
>>> Derek,
>>> Do we have traceroutes from all users? Does anything in VCenter show any 
>>> system resource constraints?
>>> 
>>> On Thursday, April 4, 2019, Person, Arthur A. <aap1@xxxxxxx> wrote:
>>> Yeh, definitely looks "blipier" starting around 7Z this morning, but 
>>> nothing like it was before.  And all last night was clean.  Here's our 
>>> graph with a 2-way split, a huge improvement over what it was before the 
>>> switch to Boulder:
>>> 
>>> <pastedImage.png>
>>> Agree with Pete that this morning's data probably isn't a good test since 
>>> there were other factors.  Since this seems so much better, I'm going to 
>>> try switching to no split as an experiment and see how it holds up.
>>> 
>>>                         Art
>>> 
>>> Arthur A. Person
>>> Assistant Research Professor, System Administrator
>>> Penn State Department of Meteorology and Atmospheric Science
>>> email:  aap1@xxxxxxx, phone:  814-863-1563
>>> 
>>> 
>>>  
>>> From: Pete Pokrandt <poker@xxxxxxxxxxxx>
>>> Sent: Thursday, April 4, 2019 1:51 PM
>>> To: Derek VanPelt - NOAA Affiliate
>>> Cc: Person, Arthur A.; Gilbert Sebenste; Anne Myckow - NOAA Affiliate; 
>>> conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; 
>>> support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [Ncep.list.pmb-dataflow] [conduit] Large lags on CONDUIT feed 
>>> - started a week or so ago
>>>  
>>> Ah, so perhaps not a good test.. I'll set it back to a 5-way split and see 
>>> how it looks tomorrow.
>>> 
>>> Thanks for the info,
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: Derek VanPelt - NOAA Affiliate <derek.vanpelt@xxxxxxxx>
>>> Sent: Thursday, April 4, 2019 12:38 PM
>>> To: Pete Pokrandt
>>> Cc: Person, Arthur A.; Gilbert Sebenste; Anne Myckow - NOAA Affiliate; 
>>> conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; 
>>> support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [Ncep.list.pmb-dataflow] [conduit] Large lags on CONDUIT feed 
>>> - started a week or so ago
>>>  
>>> HI Pete -- we did have a separate issu hit the CONDUIT feed today.  We 
>>> should be recovering now, but the backlog was sizeable.  If these numbers 
>>> are not back to the baseline in the next hour or so please let us know.  We 
>>> are also watching our queues and they are decreasing, but not as quickly as 
>>> we had hoped.
>>> 
>>> Thank you,
>>> 
>>> Derek
>>> 
>>> On Thu, Apr 4, 2019 at 1:26 PM 'Pete Pokrandt' via _NCEP list.pmb-dataflow 
>>> <ncep.list.pmb-dataflow@xxxxxxxx> wrote:
>>> FYI - there is still a much larger lag for the 12 UTC run with a 5-way 
>>> split compared to a 10-way split. It's better since everything else failed 
>>> over to Boulder, but I'd venture to guess that's not the root of the 
>>> problem.
>>> 
>>> <iddstats_CONDUIT_idd_aos_wisc_edu_ending_20190404_17UTC.gif>
>>> 
>>> Prior to whatever is going on to cause this, I don'r recall ever seeing 
>>> lags this large with a 5-way split. It looked much more like the left hand 
>>> side of this graph, with small increases in lag with each 6 hourly model 
>>> run cycle, but more like 100 seconds  vs the ~900 that I got this morning.
>>> 
>>> FYI I am going to change back to a 10 way split for now.
>>> 
>>> Pete
>>> 
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> 
>>> on behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
>>> Sent: Wednesday, April 3, 2019 4:57 PM
>>> To: Person, Arthur A.; Gilbert Sebenste; Anne Myckow - NOAA Affiliate
>>> Cc: conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; 
>>> support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed 
>>> - started a week or so ago
>>>  
>>> Sorry, was out this morning and just had a chance to look into this. I 
>>> concur with Art and Gilbert that things appear to have gotten better 
>>> starting with the failover of everything else to Boulder yesterday. I will 
>>> also reconfigure to go back to a 5-way split (as opposed to the 10-way 
>>> split that I've been using since this issue began) and keep an eye on 
>>> tomorrow's 12 UTC model run cycle - if the lags go up, it usually happens 
>>> worst during that cycle, shortly before 18 UTC each day. 
>>> 
>>> I'll report back tomorrow how it looks, or you can see at 
>>> 
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>> 
>>> Thanks,
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> 
>>> on behalf of Person, Arthur A. <aap1@xxxxxxx>
>>> Sent: Wednesday, April 3, 2019 4:04 PM
>>> To: Gilbert Sebenste; Anne Myckow - NOAA Affiliate
>>> Cc: conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; 
>>> support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed 
>>> - started a week or so ago
>>>  
>>> Anne,
>>> 
>>> I'll hop back in the loop here... for some reason these replies started 
>>> going into my junk file (bleh).  Anyway, I agree with Gilbert's assessment. 
>>>  Things turned real clean around 12Z yesterday, looking at the graphs.  I 
>>> usually look at flood.atmos.uiuc.edu when there are problem as their 
>>> connection always seems to be the cleanest.  If there are even small blips 
>>> or ups and downs in their latencies, that usually means there's a network 
>>> aberration somewhere that usually amplifies into hundreds or thousands of 
>>> seconds at our site and elsewhere.  Looking at their graph now, you can see 
>>> the blipiness up until 12Z yesterday, and then it's flat (except for the 
>>> one spike around 16Z today which I would ignore):
>>> 
>>> <pastedImage.png>
>>> Our direct-connected site, which is using a 10-way split right now, also 
>>> shows a return to calmness in the latencies:
>>> <pastedImage.png>
>>> Prior to the recent latency jump, I did not use split requests and the 
>>> reception had been stellar for quite some time.  It's my suspicion that 
>>> this is a networking congestion issue somewhere close to the source since 
>>> it seems to affect all downstream sites.  For that reason, I don't think 
>>> solving this problem should necessarily involve upgrading your server 
>>> software, but rather identifying what's jamming up the network near D.C., 
>>> and testing this by switching to Boulder was an excellent idea.  I will now 
>>> try switching our system to a two-way split to see if this performance 
>>> holds up with fewer pipes.  Thanks for your help and I'll let you know what 
>>> I find out.
>>> 
>>>                                  Art
>>> 
>>> Arthur A. Person
>>> Assistant Research Professor, System Administrator
>>> Penn State Department of Meteorology and Atmospheric Science
>>> email:  aap1@xxxxxxx, phone:  814-863-1563
>>> 
>>> 
>>>  
>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> 
>>> on behalf of Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
>>> Sent: Wednesday, April 3, 2019 4:07 PM
>>> To: Anne Myckow - NOAA Affiliate
>>> Cc: conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; 
>>> support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed 
>>> - started a week or so ago
>>>  
>>> Hello Anne,
>>> 
>>> I'll jump in here as well. Consider the CONDUIT delays at UNIDATA:
>>> 
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
>>>  
>>> 
>>> And now, Wisconsin: 
>>> 
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>>  
>>> 
>>> And finally, the University of Washington:
>>> 
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+freshair1.atmos.washington.edu
>>>   
>>> 
>>> All three of whom have direct feeds from you. Flipping over to Boulder 
>>> definitely caused a major improvement. There was still a brief spike in 
>>> delay, but much shorter and minimal
>>> compared to what it was.
>>> 
>>> Gilbert
>>> 
>>> On Wed, Apr 3, 2019 at 10:03 AM Anne Myckow - NOAA Affiliate 
>>> <anne.myckow@xxxxxxxx> wrote:
>>> Hi Pete,
>>> 
>>> As of yesterday we failed almost all of our applications to our site in 
>>> Boulder (meaning away from CONDUIT). Have you noticed an improvement in 
>>> your speeds since yesterday afternoon? If so this will give us a clue that 
>>> maybe there's something interfering on our side that isn't specifically 
>>> CONDUIT, but another app that might be causing congestion. (And if it's the 
>>> same then that's a clue in the other direction.)
>>> 
>>> Thanks,
>>> Anne
>>> 
>>> On Mon, Apr 1, 2019 at 3:24 PM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>> The lag here at UW-Madison was up to 1200 seconds today, and that's with a 
>>> 10-way split feed. Whatever is causing the issue has definitely not been 
>>> resolved, and historically is worse during the work week than on the 
>>> weekends. If that helps at all.
>>> 
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: Anne Myckow - NOAA Affiliate <anne.myckow@xxxxxxxx>
>>> Sent: Thursday, March 28, 2019 4:28 PM
>>> To: Person, Arthur A.
>>> Cc: Carissa Klemmer - NOAA Federal; Pete Pokrandt; _NCEP.List.pmb-dataflow; 
>>> conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed - started 
>>> a week or so ago
>>>  
>>> Hello Art,
>>> 
>>> We will not be upgrading to version 6.13 on these systems as they are not 
>>> robust enough to support the local logging inherent in the new version.
>>> 
>>> I will check in with my team on if there are any further actions we can 
>>> take to try and troubleshoot this issue, but I fear we may be at the limit 
>>> of our ability to make this better.
>>> 
>>> Iâll let you know tomorrow where we stand. Thanks.
>>> Anne
>>> 
>>> On Mon, Mar 25, 2019 at 3:00 PM Person, Arthur A. <aap1@xxxxxxx> wrote:
>>> Carissa,
>>> 
>>> Can you report any status on this inquiry?
>>> 
>>> Thanks...          Art
>>> 
>>> Arthur A. Person
>>> Assistant Research Professor, System Administrator
>>> Penn State Department of Meteorology and Atmospheric Science
>>> email:  aap1@xxxxxxx, phone:  814-863-1563
>>> 
>>> 
>>>  
>>> From: Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>
>>> Sent: Tuesday, March 12, 2019 8:30 AM
>>> To: Pete Pokrandt
>>> Cc: Person, Arthur A.; conduit@xxxxxxxxxxxxxxxx; 
>>> support-conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow
>>> Subject: Re: Large lags on CONDUIT feed - started a week or so ago
>>>  
>>> Hi Everyone
>>> 
>>> Iâve added the Dataflow team email to the thread. I havenât heard that any 
>>> changes were made or that any issues were found. But the team can look 
>>> today and see if we have any signifiers of overall slowness with anything. 
>>> 
>>> Dataflow, try taking a look at the new Citrix or VM troubleshooting tools 
>>> if there are any abnormal signatures that may explain this. 
>>> 
>>> On Monday, March 11, 2019, Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>> Art,
>>> 
>>> I don't know if NCEP ever figured anything out, but I've been able to keep 
>>> my latencies reasonable (300-600s max, mostly during the 12 UTC model 
>>> suite) by splitting my CONDUIT request 10 ways, instead of the 5 that I had 
>>> been doing, or in a single request. Maybe give that a try and see if it 
>>> helps at all.
>>> 
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: Person, Arthur A. <aap1@xxxxxxx>
>>> Sent: Monday, March 11, 2019 3:45 PM
>>> To: Holly Uhlenhake - NOAA Federal; Pete Pokrandt
>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or so ago
>>>  
>>> Holly,
>>> 
>>> Was there any resolution to this on the NCEP end?  I'm still seeing 
>>> terrible delays (1000-4000 seconds) receiving data from 
>>> conduit.ncep.noaa.gov.  It would be helpful to know if things are resolved 
>>> at NCEP's end so I know whether to look further down the line.
>>> 
>>> Thanks...           Art
>>> 
>>> Arthur A. Person
>>> Assistant Research Professor, System Administrator
>>> Penn State Department of Meteorology and Atmospheric Science
>>> email:  aap1@xxxxxxx, phone:  814-863-1563
>>> 
>>> 
>>>  
>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> 
>>> on behalf of Holly Uhlenhake - NOAA Federal <holly.uhlenhake@xxxxxxxx>
>>> Sent: Thursday, February 21, 2019 12:05 PM
>>> To: Pete Pokrandt
>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or so ago
>>>  
>>> Hi Pete,
>>> 
>>> We'll take a look and see if we can figure out what might be going on.  We 
>>> haven't done anything to try and address this yet, but based on your 
>>> analysis I'm suspicious that it might be tied to a resource constraint on 
>>> the VM or the blade it resides on.
>>> 
>>> Thanks,
>>> Holly Uhlenhake
>>> Acting Dataflow Team Lead 
>>> 
>>> On Thu, Feb 21, 2019 at 11:32 AM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>> Just FYI, data is flowing, but the large lags continue.
>>> 
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
>>> 
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> 
>>> on behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
>>> Sent: Wednesday, February 20, 2019 12:07 PM
>>> To: Carissa Klemmer - NOAA Federal
>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or so ago
>>>  
>>> Data is flowing again - picked up somewhere in the GEFS. Maybe CONDUIT 
>>> server was restarted, or ldm on it? Lags are large (3000s+) but dropping 
>>> slowly
>>> 
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> 
>>> on behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
>>> Sent: Wednesday, February 20, 2019 11:56 AM
>>> To: Carissa Klemmer - NOAA Federal
>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or so ago
>>>  
>>> Just a quick follow-up - we started falling far enough behind (3600+ sec) 
>>> that we are losing data. We got short files starting at 174h into the GFS 
>>> run, and only got (incomplete) data through 207h.
>>> 
>>> We have now not received any data on CONDUIT since 11:27 AM CST (1727 UTC) 
>>> today (Wed Feb 20)
>>> 
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> 
>>>  
>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx> 
>>> on behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
>>> Sent: Wednesday, February 20, 2019 11:28 AM
>>> To: Carissa Klemmer - NOAA Federal
>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>> Subject: [conduit] Large lags on CONDUIT feed - started a week or so ago
>>>  
>>> Carissa,
>>> 
>>> We have been feeding CONDUIT using a 5 way split feed direct from 
>>> conduit.ncep.noaa.gov, and it had been really good for some time, lags 
>>> 30-60 seconds or less.
>>> 
>>> However, the past week or so, we've been seeing some very large lags during 
>>> each 6 hour model suite - Unidata is also seeing these - they are also 
>>> feeding direct from conduit.ncep.noaa.gov.
>>> 
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>> 
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
>>> 
>>> 
>>> Any idea what's going on, or how we can find out? 
>>> 
>>> Thanks!
>>> Pete
>>> 
>>> 
>>> --
>>> Pete Pokrandt - Systems Programmer
>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>> 608-262-3086  - poker@xxxxxxxxxxxx
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web.  Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>> 
>>> 
>>> conduit mailing list
>>> conduit@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/
>>> 
>>> 
>>> -- 
>>> Carissa Klemmer 
>>> NCEP Central Operations 
>>> IDSB Branch Chief
>>> 301-683-3835
>>> 
>>> _______________________________________________
>>> Ncep.list.pmb-dataflow mailing list
>>> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>>> -- 
>>> Anne Myckow
>>> Lead Dataflow Analyst
>>> NOAA/NCEP/NCO
>>> 301-683-3825
>>> 
>>> 
>>> -- 
>>> Anne Myckow
>>> Lead Dataflow Analyst
>>> NOAA/NCEP/NCO
>>> 301-683-3825
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web.  Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>> 
>>> 
>>> conduit mailing list
>>> conduit@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/
>>> 
>>> 
>>> -- 
>>> ----
>>>  
>>> Gilbert Sebenste
>>> Consulting Meteorologist
>>> AllisonHouse, LLC
>>> _______________________________________________
>>> Ncep.list.pmb-dataflow mailing list
>>> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>>> 
>>> 
>>> -- 
>>> Derek Van Pelt
>>> DataFlow Analyst
>>> NOAA/NCEP/NCO
>>> 
>>> 
>>> -- 
>>> Carissa Klemmer 
>>> NCEP Central Operations 
>>> IDSB Branch Chief
>>> 301-683-3835
>>> 
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web.  Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>> 
>>> 
>>> conduit mailing list
>>> conduit@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/
>>> 
>>> 
>>> -- 
>>> Derek Van Pelt
>>> DataFlow Analyst
>>> NOAA/NCEP/NCO
>>> -- 
>>> Misspelled straight from Derek's phone.
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web.  Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>> 
>>> 
>>> conduit mailing list
>>> conduit@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/
>>> _______________________________________________
>>> Ncep.list.pmb-dataflow mailing list
>>> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>>> 
>>> 
>>> -- 
>>> Dustin Sheffler
>>> NCEP Central Operations - Dataflow
>>> 5830 University Research Court, Rm 1030
>>> College Park, Maryland 20740
>>> Office: (301) 683-3827
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web.  Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>> 
>>> 
>>> conduit mailing list
>>> conduit@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/