Hi Anne,
I can answer. Not as good as before, but still much better through 13:27Z:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
Gilbert
> On Jul 3, 2019, at 8:17 AM, Anne Myckow - NOAA Affiliate
> <anne.myckow@xxxxxxxx> wrote:
>
> Pete et al,
>
> Can you tell us how the latency looks this morning and overnight?
>
> Thanks,
> Anne
>
>> On Tue, Jul 2, 2019 at 9:23 PM Anne Myckow - NOAA Affiliate
>> <anne.myckow@xxxxxxxx> wrote:
>> Hi Pete,
>>
>> We've been able to re-create the CONDUIT LDM issues with other LDMs now in
>> NCO. We do not know root cause but we are failing some services out of
>> College Park now to alleviate the traffic. You may experience slowness again
>> tomorrow while we troubleshoot with the whole team in office but overnight
>> (Eastern Time anyway) should be better.
>>
>> I'm adding you and the other people with actual email addresses (rather than
>> the lists) to the email chain where we are keeping everyone apprised, so
>> don't be surprised to get another email that says OPEN: TID <lots of other
>> text> in the subject line - that's about this slowness.
>>
>> Thanks,
>> Anne
>>
>>> On Tue, Jul 2, 2019 at 11:49 AM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>> Thanks, Anne.
>>>
>>> Lag is still there on the current 12 UTC cycle FYI
>>>
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>>
>>> Pete
>>>
>>> Sent from my iPhone
>>>
>>> On Jul 2, 2019, at 10:18 AM, Anne Myckow - NOAA Affiliate
>>> <anne.myckow@xxxxxxxx> wrote:
>>>
>>>> Hi Pete,
>>>>
>>>> We (NCO) have fully loaded our College Park site again, where conduit
>>>> lives. I'll see if I can get the attention of our networking folks today
>>>> about this since they just installed new hardware that we believe should
>>>> have increased our network capacity.
>>>>
>>>> Thanks,
>>>> Anne
>>>>
>>>>> On Tue, Jul 2, 2019 at 1:25 AM 'Pete Pokrandt' via _NCEP
>>>>> list.pmb-dataflow <ncep.list.pmb-dataflow@xxxxxxxx> wrote:
>>>>> All,
>>>>>
>>>>> Something happened in the past day or two that has resulted in large lags
>>>>> (and data loss) between conduit.ncep.noaa.gov and idd.aos.wisc.edu (and
>>>>> Unidata too)
>>>>>
>>>>> Based on these IDD stats, there was a bit of a lag increase with the 06
>>>>> UTC July 1 runs, a little larger with the 12 UTC runs, and then much
>>>>> bigger for the 18 UTC July 1 and 00 UTC July 2 runs. Any idea what might
>>>>> have happened or changed? The fact that Unidata's and UW-AOS's graphs
>>>>> look so similar suggests that it's something upstream of us.
>>>>>
>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>>>>
>>>>> <iddstats_conduit_idd_aos_wisc_edu_20190702.gif>
>>>>>
>>>>> Here's Unidata's graph:
>>>>>
>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+lead.unidata.ucar.edu
>>>>>
>>>>> <iddstats_conduit_lead_unidata_ucar_edu_20190702.gif>
>>>>>
>>>>> Thanks,
>>>>> Pete
>>>>>
>>>>>
>>>>> --
>>>>> Pete Pokrandt - Systems Programmer
>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>
>>>>>
>>>>> From: Derek VanPelt - NOAA Affiliate <derek.vanpelt@xxxxxxxx>
>>>>> Sent: Tuesday, April 23, 2019 3:40 PM
>>>>> To: Pete Pokrandt
>>>>> Cc: Person, Arthur A.; Gilbert Sebenste; Kevin Goebbert;
>>>>> conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; Mike Zuranski; Dustin
>>>>> Sheffler - NOAA Federal; support-conduit@xxxxxxxxxxxxxxxx
>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>> feed - started a week or so ago
>>>>>
>>>>> Hi All,
>>>>>
>>>>> There are few things going on here.
>>>>>
>>>>> The strongest driver on you download speeds is the presence or absence of
>>>>> NOMADS in College Park. When NOMADS is in CPRK, dissemination from the
>>>>> entire datacenter (including our Conduit servers which only exist in
>>>>> College Park) can be effected at peak model download times. Adding to
>>>>> this are new rules put in place that require the NOMADS users to all
>>>>> follow the top level VIP. Previously some of our users would pull from
>>>>> Boulder even when the VIP pointed to College Park. That is no longer
>>>>> regularly possible, as the backup server is intentionally being blocked
>>>>> to traffic.
>>>>>
>>>>> I have been asked to go back and using internal metrics and the download
>>>>> speeds that have been provided in this thread (thanks!) to firmly
>>>>> establish the time line, and hope to do so in the next few days, but
>>>>> believe the answer will be as stated above.
>>>>>
>>>>> As far as splitting the request into many smaller requests; it clearly is
>>>>> having a positive effect. As long as you don't (and we don't) hit an
>>>>> upper connection count limit, this appears to be the best way to minimize
>>>>> the latency during peak download times.
>>>>>
>>>>> More to come. Thanks for keeping this discussion alive as it has
>>>>> provided light for both the Conduit download speeds, but also provides
>>>>> context for some of our wide ranging issues.
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Derek
>>>>>
>>>>> On Tue, Apr 23, 2019 at 3:07 PM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>>>> I'm still on the 10 way split that I've been on for quite some time, and
>>>>> without my changing anything, our lags got much much better starting on
>>>>> Friday, 4/19 starting with the 12 UTC model sequence. I don't know if
>>>>> this correlated to Unidata switching to a 20 way split or not, but that
>>>>> happened around the same time.
>>>>>
>>>>> Here are my lag plots, the first ends 04 UTC 4/20, and the second just
>>>>> now at 19 UTC 4/23. Note the Y axis on the first plot goes to ~3600
>>>>> seconds, but on the second plot, only to ~100 seconds.
>>>>>
>>>>>
>>>>>
>>>>> <iddstats_CONDUIT_idd_aos_wisc_edu_ending_20190423_1900UTC.gif>
>>>>>
>>>>> Pete
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Pete Pokrandt - Systems Programmer
>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>
>>>>>
>>>>> From: Person, Arthur A. <aap1@xxxxxxx>
>>>>> Sent: Tuesday, April 23, 2019 1:49 PM
>>>>> To: Pete Pokrandt; Gilbert Sebenste
>>>>> Cc: Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>> Mike Zuranski; Derek VanPelt - NOAA Affiliate; Dustin Sheffler - NOAA
>>>>> Federal; support-conduit@xxxxxxxxxxxxxxxx
>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>> feed - started a week or so ago
>>>>>
>>>>> I switched our test system iddrs2a feeding from conduit.ncep.noaa.gov
>>>>> back to a 2-way split (from a 20-way split) yesterday to see how it would
>>>>> hold up:
>>>>>
>>>>> <pastedImage.png>
>>>>>
>>>>> While not as good as prior to February, it wasn't terrible, at least
>>>>> until this morning. Looks like the 20-way split may be the solution
>>>>> going forward if this is the "new normal" for network performance.
>>>>>
>>>>> Art
>>>>>
>>>>>
>>>>> Arthur A. Person
>>>>> Assistant Research Professor, System Administrator
>>>>> Penn State Department of Meteorology and Atmospheric Science
>>>>> email: aap1@xxxxxxx, phone: 814-863-1563
>>>>>
>>>>>
>>>>>
>>>>> From: Pete Pokrandt <poker@xxxxxxxxxxxx>
>>>>> Sent: Saturday, April 20, 2019 12:29 AM
>>>>> To: Person, Arthur A.; Gilbert Sebenste
>>>>> Cc: Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>> Mike Zuranski; Derek VanPelt - NOAA Affiliate; Dustin Sheffler - NOAA
>>>>> Federal; support-conduit@xxxxxxxxxxxxxxxx
>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>> feed - started a week or so ago
>>>>>
>>>>> Well, I haven't changed anything in the past few days, but my lags
>>>>> dropped back to pretty much pre-February 10 levels starting with today's
>>>>> (20190419) 12 UTC run. I know Unidata switched to a 20 way split feed
>>>>> around that same time... I am still running a 10-way split. I didn't
>>>>> change anything between today's 06 UTC run and the 12 UTC run, but the
>>>>> lags dropped considerably, and look like they used to.
>>>>>
>>>>> I wonder if some bad piece of hardware got swapped out somewhere, or if
>>>>> some change was made internally at NCEP that fixed whatever was going on.
>>>>> Or, perhaps the Unidata switch to a 20 way feed somehow reduced a load on
>>>>> a router somewhere and data is getting through more easily?
>>>>>
>>>>> Strange..
>>>>>
>>>>> Pete
>>>>>
>>>>> <conduit_lag_idd.aos.wisc.edu_20180420_0409UTC.gif>
>>>>>
>>>>> --
>>>>> Pete Pokrandt - Systems Programmer
>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>
>>>>>
>>>>> From: Person, Arthur A. <aap1@xxxxxxx>
>>>>> Sent: Thursday, April 18, 2019 2:20 PM
>>>>> To: Gilbert Sebenste; Pete Pokrandt
>>>>> Cc: Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>> Mike Zuranski; Derek VanPelt - NOAA Affiliate; Dustin Sheffler - NOAA
>>>>> Federal; support-conduit@xxxxxxxxxxxxxxxx
>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>> feed - started a week or so ago
>>>>>
>>>>> All --
>>>>>
>>>>> I switched our test system, iddrs2a, feeding from conduit.ncep.noaa.gov
>>>>> from a 2-way split to a 20-way split yesterday, and the results are
>>>>> dramatic:
>>>>>
>>>>> <pastedImage.png>
>>>>> Although conduit feed performance at other sites improved a little last
>>>>> night with the MRMS feed failure, it doesn't explain this improvement
>>>>> entirely. This leads me to ponder the causes of such an improvement:
>>>>>
>>>>> 1) The network path does not appear to be bandwidth constrained,
>>>>> otherwise there would be no improvement no matter how many pipes were
>>>>> used;
>>>>>
>>>>> 2) The problem, therefore, would appear to be packet oriented, either
>>>>> with path packet saturation, or packet shaping.
>>>>>
>>>>> I'm not a networking expert, so maybe I'm missing another possibility
>>>>> here, but I'm curious whether packet shaping could account for some of
>>>>> the throughput issues. I've also been having trouble getting timely
>>>>> delivery of our Unidata IDD satellite feed, and discovered that switching
>>>>> that to a 10-way split feed (from a 2-way split) has reduced the
>>>>> latencies from 2000-3000 seconds down to less than 300 seconds.
>>>>> Interestingly, the peak satellite feed latencies (see below) occur at the
>>>>> same time as the peak conduit latencies, but this path is unrelated to
>>>>> NCEP (as far as I know). Is it possible that Internet 2 could be
>>>>> packet-shaping their traffic and that this could be part of the cause for
>>>>> the packet latencies we're seeing?
>>>>>
>>>>> Art
>>>>>
>>>>> <pastedImage.png>
>>>>>
>>>>>
>>>>> Arthur A. Person
>>>>> Assistant Research Professor, System Administrator
>>>>> Penn State Department of Meteorology and Atmospheric Science
>>>>> email: aap1@xxxxxxx, phone: 814-863-1563
>>>>>
>>>>>
>>>>>
>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx>
>>>>> on behalf of Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
>>>>> Sent: Thursday, April 18, 2019 2:29 AM
>>>>> To: Pete Pokrandt
>>>>> Cc: Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>> Mike Zuranski; Derek VanPelt - NOAA Affiliate; Dustin Sheffler - NOAA
>>>>> Federal; support-conduit@xxxxxxxxxxxxxxxx
>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>> feed - started a week or so ago
>>>>>
>>>>> FYI: all evening and into the overnight, MRMS data has been missing, QC
>>>>> BR has been town for the last 40 minutes, but smaller products are coming
>>>>> through somewhat more reliably as of 6Z. CONDUIT was still substantially
>>>>> delayed around 4Z with the GFS.
>>>>>
>>>>> Gilbert
>>>>>
>>>>> On Apr 16, 2019, at 5:43 PM, Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>>>>
>>>>>> Here's a few traceroutes from just now - from idd-agg.aos.wisc.edu to
>>>>>> conduit.ncep.noaa.gov. The lags are up and running around 600-800
>>>>>> seconds right now. I'm not including all of the * * * lines from after
>>>>>> 144.90.76.65 which is presumably behind a firewall.
>>>>>>
>>>>>>
>>>>>> 2209 UTC Tuesday Apr 16
>>>>>>
>>>>>> traceroute -p 388 conduit.ncep.noaa.gov
>>>>>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60
>>>>>> byte packets
>>>>>> 1 vlan-510-cssc-gw.net.wisc.edu (144.92.130.1) 0.906 ms 0.701 ms
>>>>>> 0.981 ms
>>>>>> 2 128.104.4.129 (128.104.4.129) 1.700 ms 1.737 ms 1.772 ms
>>>>>> 3 rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)
>>>>>> 1.740 ms 3.343 ms 3.336 ms
>>>>>> 4 rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu
>>>>>> (146.151.166.122) 2.043 ms 2.034 ms 1.796 ms
>>>>>> 5 144.92.254.229 (144.92.254.229) 11.530 ms 11.472 ms 11.535 ms
>>>>>> 6 et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60) 22.813
>>>>>> ms 22.899 ms 22.886 ms
>>>>>> 7 et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2) 24.248 ms
>>>>>> 24.195 ms 24.172 ms
>>>>>> 8 nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189) 24.244 ms
>>>>>> 24.196 ms 24.183 ms
>>>>>> 9 ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4) 24.937 ms 24.884 ms
>>>>>> 24.878 ms
>>>>>> 10 140.208.63.30 (140.208.63.30) 134.030 ms 126.195 ms 126.305 ms
>>>>>> 11 140.90.76.65 (140.90.76.65) 106.810 ms 104.553 ms 104.603 ms
>>>>>>
>>>>>> 2230 UTC Tuesday Apr 16
>>>>>>
>>>>>> traceroute -p 388 conduit.ncep.noaa.gov
>>>>>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60
>>>>>> byte packets
>>>>>> 1 vlan-510-cssc-gw.net.wisc.edu (144.92.130.1) 1.391 ms 1.154 ms
>>>>>> 5.902 ms
>>>>>> 2 128.104.4.129 (128.104.4.129) 6.917 ms 6.895 ms 2.004 ms
>>>>>> 3 rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)
>>>>>> 3.158 ms 3.293 ms 3.251 ms
>>>>>> 4 rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu
>>>>>> (146.151.166.122) 6.185 ms 2.278 ms 2.425 ms
>>>>>> 5 144.92.254.229 (144.92.254.229) 6.909 ms 13.255 ms 6.863 ms
>>>>>> 6 et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60) 23.328
>>>>>> ms 23.244 ms 28.845 ms
>>>>>> 7 et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2) 30.308 ms
>>>>>> 24.575 ms 24.536 ms
>>>>>> 8 nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189) 29.594 ms
>>>>>> 24.624 ms 24.618 ms
>>>>>> 9 ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4) 24.581 ms 30.164 ms
>>>>>> 24.627 ms
>>>>>> 10 140.208.63.30 (140.208.63.30) 25.677 ms 25.767 ms 29.543 ms
>>>>>> 11 140.90.76.65 (140.90.76.65) 105.812 ms 105.345 ms 108.857
>>>>>>
>>>>>> 2232 UTC Tuesday Apr 16
>>>>>>
>>>>>> traceroute -p 388 conduit.ncep.noaa.gov
>>>>>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60
>>>>>> byte packets
>>>>>> 1 vlan-510-cssc-gw.net.wisc.edu (144.92.130.1) 1.266 ms 1.070 ms
>>>>>> 1.226 ms
>>>>>> 2 128.104.4.129 (128.104.4.129) 1.915 ms 2.652 ms 2.775 ms
>>>>>> 3 rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)
>>>>>> 2.353 ms 2.129 ms 2.314 ms
>>>>>> 4 rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu
>>>>>> (146.151.166.122) 2.114 ms 2.111 ms 2.163 ms
>>>>>> 5 144.92.254.229 (144.92.254.229) 6.891 ms 6.838 ms 6.840 ms
>>>>>> 6 et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60) 23.336
>>>>>> ms 23.283 ms 23.364 ms
>>>>>> 7 et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2) 24.493 ms
>>>>>> 24.136 ms 24.152 ms
>>>>>> 8 nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189) 24.161 ms
>>>>>> 24.173 ms 24.176 ms
>>>>>> 9 ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4) 24.165 ms 24.331 ms
>>>>>> 24.201 ms
>>>>>> 10 140.208.63.30 (140.208.63.30) 25.361 ms 25.427 ms 25.240 ms
>>>>>> 11 140.90.76.65 (140.90.76.65) 113.194 ms 115.553 ms 115.543 ms
>>>>>>
>>>>>>
>>>>>> 2234 UTC Tuesday Apr 16
>>>>>>
>>>>>> traceroute -p 388 conduit.ncep.noaa.gov
>>>>>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60
>>>>>> byte packets
>>>>>> 1 vlan-510-cssc-gw.net.wisc.edu (144.92.130.1) 0.901 ms 0.663 ms
>>>>>> 0.826 ms
>>>>>> 2 128.104.4.129 (128.104.4.129) 1.645 ms 1.948 ms 1.729 ms
>>>>>> 3 rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)
>>>>>> 1.804 ms 1.788 ms 1.849 ms
>>>>>> 4 rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu
>>>>>> (146.151.166.122) 2.011 ms 2.004 ms 1.982 ms
>>>>>> 5 144.92.254.229 (144.92.254.229) 6.241 ms 6.240 ms 6.220 ms
>>>>>> 6 et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60) 23.042
>>>>>> ms 23.072 ms 23.033 ms
>>>>>> 7 et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2) 24.094 ms
>>>>>> 24.398 ms 24.370 ms
>>>>>> 8 nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189) 24.166 ms
>>>>>> 24.166 ms 24.108 ms
>>>>>> 9 ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4) 24.056 ms 24.306 ms
>>>>>> 24.215 ms
>>>>>> 10 140.208.63.30 (140.208.63.30) 25.199 ms 25.284 ms 25.351 ms
>>>>>> 11 140.90.76.65 (140.90.76.65) 118.314 ms 118.707 ms 118.768 ms
>>>>>>
>>>>>> 2236 UTC Tuesday Apr 16
>>>>>>
>>>>>> traceroute -p 388 conduit.ncep.noaa.gov
>>>>>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60
>>>>>> byte packets
>>>>>> 1 vlan-510-cssc-gw.net.wisc.edu (144.92.130.1) 0.918 ms 0.736 ms
>>>>>> 0.864 ms
>>>>>> 2 128.104.4.129 (128.104.4.129) 1.517 ms 1.630 ms 1.734 ms
>>>>>> 3 rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)
>>>>>> 1.998 ms 3.437 ms 3.437 ms
>>>>>> 4 rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu
>>>>>> (146.151.166.122) 1.899 ms 1.896 ms 1.867 ms
>>>>>> 5 144.92.254.229 (144.92.254.229) 6.384 ms 6.317 ms 6.314 ms
>>>>>> 6 et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60) 22.980
>>>>>> ms 23.167 ms 23.078 ms
>>>>>> 7 et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2) 24.181 ms
>>>>>> 24.152 ms 24.121 ms
>>>>>> 8 nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189) 48.556 ms
>>>>>> 47.824 ms 47.799 ms
>>>>>> 9 ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4) 24.166 ms 24.154 ms
>>>>>> 24.214 ms
>>>>>> 10 140.208.63.30 (140.208.63.30) 25.310 ms 25.268 ms 25.401 ms
>>>>>> 11 140.90.76.65 (140.90.76.65) 118.299 ms 123.763 ms 122.207 ms
>>>>>>
>>>>>> 2242 UTC
>>>>>>
>>>>>> traceroute -p 388 conduit.ncep.noaa.gov
>>>>>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60
>>>>>> byte packets
>>>>>> 1 vlan-510-cssc-gw.net.wisc.edu (144.92.130.1) 1.337 ms 1.106 ms
>>>>>> 1.285 ms
>>>>>> 2 128.104.4.129 (128.104.4.129) 6.039 ms 5.778 ms 1.813 ms
>>>>>> 3 rx-cssc-b380-1-core-bundle-ether2-1521.net.wisc.edu (146.151.168.4)
>>>>>> 2.275 ms 2.464 ms 2.517 ms
>>>>>> 4 rx-animal-226-2-core-bundle-ether1-1928.net.wisc.edu
>>>>>> (146.151.166.122) 2.288 ms 6.978 ms 3.506 ms
>>>>>> 5 144.92.254.229 (144.92.254.229) 10.369 ms 6.626 ms 10.281 ms
>>>>>> 6 et-1-1-5.4079.rtsw.ashb.net.internet2.edu (162.252.70.60) 23.513
>>>>>> ms 23.297 ms 23.295 ms
>>>>>> 7 et-11-3-0-1275.clpk-core.maxgigapop.net (206.196.177.2) 27.938 ms
>>>>>> 24.589 ms 28.783 ms
>>>>>> 8 nwave-clpk-re.demarc.maxgigapop.net (206.196.177.189) 28.796 ms
>>>>>> 24.630 ms 28.793 ms
>>>>>> 9 ae-2.666.rtr.clpk.nwave.noaa.gov (137.75.68.4) 24.576 ms 24.545 ms
>>>>>> 24.587 ms
>>>>>> 10 140.208.63.30 (140.208.63.30) 85.763 ms 85.768 ms 83.623 ms
>>>>>> 11 140.90.76.65 (140.90.76.65) 131.912 ms 132.662 ms 132.340 ms
>>>>>>
>>>>>> Pete
>>>>>>
>>>>>> --
>>>>>> Pete Pokrandt - Systems Programmer
>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>
>>>>>>
>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt
>>>>>> <poker@xxxxxxxxxxxx>
>>>>>> Sent: Tuesday, April 16, 2019 3:04 PM
>>>>>> To: Gilbert Sebenste; Tyle, Kevin R
>>>>>> Cc: Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>>> Derek VanPelt - NOAA Affiliate; Mike Zuranski; Dustin Sheffler - NOAA
>>>>>> Federal; support-conduit@xxxxxxxxxxxxxxxx
>>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>>> feed - started a week or so ago
>>>>>>
>>>>>> At UW-Madison, we had incomplete 12 UTC GFS data starting with the 177h
>>>>>> forecast. Lags exceeded 3600s.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Pete
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Pete Pokrandt - Systems Programmer
>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>
>>>>>>
>>>>>> From: Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
>>>>>> Sent: Tuesday, April 16, 2019 2:44 PM
>>>>>> To: Tyle, Kevin R
>>>>>> Cc: Pete Pokrandt; Dustin Sheffler - NOAA Federal; Mike Zuranski; Derek
>>>>>> VanPelt - NOAA Affiliate; Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx;
>>>>>> _NCEP.List.pmb-dataflow; support-conduit@xxxxxxxxxxxxxxxx
>>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>>> feed - started a week or so ago
>>>>>>
>>>>>> Yes, here at AllisonHouse too...we can feed from a number of sites, and
>>>>>> all of them were dropping GFS, and delayed by an hour.
>>>>>>
>>>>>> Gilbert
>>>>>>
>>>>>> On Apr 16, 2019, at 2:39 PM, Tyle, Kevin R <ktyle@xxxxxxxxxx> wrote:
>>>>>>
>>>>>>> For what it's worth, our 12Z GFS data ingest was quite bad today ...
>>>>>>> many lost products beyond F168 (we feed from UWisc-MSN primary and PSU
>>>>>>> secondary).
>>>>>>>
>>>>>>> _____________________________________________
>>>>>>> Kevin Tyle, M.S.; Manager of Departmental Computing
>>>>>>> Dept. of Atmospheric & Environmental Sciences
>>>>>>> University at Albany
>>>>>>> Earth Science 235, 1400 Washington Avenue
>>>>>>> Albany, NY 12222
>>>>>>> Email: ktyle@xxxxxxxxxx
>>>>>>> Phone: 518-442-4578
>>>>>>> _____________________________________________
>>>>>>>
>>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt
>>>>>>> <poker@xxxxxxxxxxxx>
>>>>>>> Sent: Tuesday, April 16, 2019 12:00 PM
>>>>>>> To: Dustin Sheffler - NOAA Federal; Mike Zuranski
>>>>>>> Cc: Kevin Goebbert; conduit@xxxxxxxxxxxxxxxx; Derek VanPelt - NOAA
>>>>>>> Affiliate; _NCEP.List.pmb-dataflow; support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>>>> feed - started a week or so ago
>>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> Just keeping this in the foreground.
>>>>>>>
>>>>>>> CONDUIT lags continue to be very large compared to what they were
>>>>>>> previous to whatever changed back in February. Prior to that, we rarely
>>>>>>> saw lags more than ~300s. Now they are routinely 1500-2000s at
>>>>>>> UW-Madison and Penn State, and over 3000s at Unidata - they appear to
>>>>>>> be on the edge of losing data. This does not bode well with all of the
>>>>>>> IDP applications failing back over to CP today..
>>>>>>>
>>>>>>> Can we send you some traceroutes and you back to us to maybe try to
>>>>>>> isolate where in the network this is happening? It feels like
>>>>>>> congestion or a bad route somewhere - the lags seem to be worse on
>>>>>>> weekdays than weekends if that helps at all.
>>>>>>>
>>>>>>> Here are the current CONDUIT lags to UW-Madison, Penn State and Unidata.
>>>>>>>
>>>>>>>
>>>>>>> <iddstats_CONDUIT_idd_aos_wisc_edu_ending_20190416_1600UTC.gif>
>>>>>>>
>>>>>>> <iddstats_CONDUIT_idd_meteo_psu_edu_ending_20190416_1600UTC.gif>
>>>>>>>
>>>>>>> <iddstats_CONDUIT_conduit_unidata_ucar_edu_ending_20190416_1600UTC.gif>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Dustin Sheffler - NOAA
>>>>>>> Federal <dustin.sheffler@xxxxxxxx>
>>>>>>> Sent: Tuesday, April 9, 2019 12:52 PM
>>>>>>> To: Mike Zuranski
>>>>>>> Cc: conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow; Derek VanPelt -
>>>>>>> NOAA Affiliate; support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>>>> feed - started a week or so ago
>>>>>>>
>>>>>>> Hi Mike,
>>>>>>>
>>>>>>> Thanks for the feedback on NOMADS. We recently found a slowness issue
>>>>>>> when NOMADS is running out of our Boulder data center that is being
>>>>>>> worked on by our teams now that NOMADS is live out of the College Park
>>>>>>> data center. It's hard sometimes to quantify whether slowness issues
>>>>>>> that are only being reported by a handful of users is a result of
>>>>>>> something wrong in our data center, a bad network path between a
>>>>>>> customer (possibly just from a particular region of the country) and
>>>>>>> our data center, a local issue on the customers' end, or any other
>>>>>>> reason that might cause slowness.
>>>>>>>
>>>>>>> Conduit is only ever run from our College Park data center. It's
>>>>>>> slowness is not tied into the Boulder NOMADS issue, but it does seem to
>>>>>>> be at least a little bit tied to which of our data centers NOMADS is
>>>>>>> running out of. When NOMADS is in Boulder along with the majority of
>>>>>>> our other NCEP applications, the strain on the College Park data center
>>>>>>> is minimal and Conduit appears to be running better as a result. When
>>>>>>> NOMADS runs in College Park (as it has since late yesterday) there is
>>>>>>> more strain on the data center and Conduit appears (based on provided
>>>>>>> user graphs) to run a bit worse around peak model times as a result.
>>>>>>> These are just my observations and we are still investigating what may
>>>>>>> have changed that caused the Conduit latencies to appear in the first
>>>>>>> place so that we can resolve this potential constraint.
>>>>>>>
>>>>>>> -Dustin
>>>>>>>
>>>>>>> On Tue, Apr 9, 2019 at 4:28 PM Mike Zuranski <zuranski@xxxxxxxxxxxxxxx>
>>>>>>> wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I've avoided jumping into this conversation since I don't deal much
>>>>>>> with Conduit these days, but Derek just mentioned something that I do
>>>>>>> have some applicable feedback on...
>>>>>>>
>>>>>>> > Two items happened last night. 1. NOMADS was moved back to College
>>>>>>> > Park...
>>>>>>>
>>>>>>> We get nearly all of our model data via NOMADS. When it switched to
>>>>>>> Boulder last week we saw a significant drop in download speeds, down to
>>>>>>> a couple hundred KB/s or slower. Starting last night, we're back to
>>>>>>> speeds on the order of MB/s or tens of MB/s. Switching back to College
>>>>>>> Park seems to confirm for me something about routing from Boulder was
>>>>>>> responsible. But again this was all on NOMADS, not sure if it's
>>>>>>> related to happenings on Conduit .
>>>>>>>
>>>>>>> When I noticed this last week I sent an email to sdm@xxxxxxxx including
>>>>>>> a traceroute taken at the time, let me know if you'd like me to find
>>>>>>> that and pass it along here or someplace else.
>>>>>>>
>>>>>>> -Mike
>>>>>>>
>>>>>>> ======================
>>>>>>> Mike Zuranski
>>>>>>> Meteorology Support Analyst
>>>>>>> College of DuPage - Nexlab
>>>>>>> Weather.cod.edu
>>>>>>> ======================
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 9, 2019 at 10:51 AM Person, Arthur A. <aap1@xxxxxxx> wrote:
>>>>>>> Derek,
>>>>>>>
>>>>>>> Do we know what change might have been made around February 10th when
>>>>>>> the CONDUIT problems first started happening? Prior to that time, the
>>>>>>> CONDUIT feed had been very crisp for a long period of time.
>>>>>>>
>>>>>>> Thanks... Art
>>>>>>>
>>>>>>>
>>>>>>> Arthur A. Person
>>>>>>> Assistant Research Professor, System Administrator
>>>>>>> Penn State Department of Meteorology and Atmospheric Science
>>>>>>> email: aap1@xxxxxxx, phone: 814-863-1563
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Derek VanPelt - NOAA Affiliate <derek.vanpelt@xxxxxxxx>
>>>>>>> Sent: Tuesday, April 9, 2019 11:34 AM
>>>>>>> To: Holly Uhlenhake - NOAA Federal
>>>>>>> Cc: Carissa Klemmer - NOAA Federal; Person, Arthur A.; Pete Pokrandt;
>>>>>>> _NCEP.List.pmb-dataflow; conduit@xxxxxxxxxxxxxxxx;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or
>>>>>>> so ago
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Two items happened last night.
>>>>>>>
>>>>>>> 1. NOMADS was moved back to College Park, which means there was a lot
>>>>>>> more traffic going out which will have effect on the Conduit latencies.
>>>>>>> We do not have a full load from the COllege Park Servers as many of
>>>>>>> the other applications are still running from Boulder, but NOMADS will
>>>>>>> certainly increase overall load.
>>>>>>>
>>>>>>> 2. As Holly said, there were further issues delaying and changing the
>>>>>>> timing of the model output yesterday afternoon/evening. I will be
>>>>>>> watching from our end, and monitoring the Unidata 48 hour graph (thank
>>>>>>> you for the link) throughout the day,
>>>>>>>
>>>>>>> Please let us know if you have questions or more information to help us
>>>>>>> analyse what you are seeing.
>>>>>>>
>>>>>>> Thank you,
>>>>>>>
>>>>>>> Derek
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 9, 2019 at 6:50 AM Holly Uhlenhake - NOAA Federal
>>>>>>> <holly.uhlenhake@xxxxxxxx> wrote:
>>>>>>> Hi Pete,
>>>>>>>
>>>>>>> We also had an issue on the supercomputer yesterday where several
>>>>>>> models going to conduit would have been stacked on top of each other
>>>>>>> instead of coming out in a more spread out fashion. It's not
>>>>>>> inconceivable that conduit could have backed up working through the
>>>>>>> abnormally large glut of grib messages. Are things better this
>>>>>>> morning at all?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Holly
>>>>>>>
>>>>>>> On Tue, Apr 9, 2019 at 12:37 AM Pete Pokrandt <poker@xxxxxxxxxxxx>
>>>>>>> wrote:
>>>>>>> Something changed starting with today's 18 UTC model cycle, and our
>>>>>>> lags shot up to over 3600 seconds, where we started losing data. They
>>>>>>> are growing again now with the 00 UTC cycle as well. PSU and Unidata
>>>>>>> CONDUIT stats show similar abnormally large lags.
>>>>>>>
>>>>>>> FYI.
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: Person, Arthur A. <aap1@xxxxxxx>
>>>>>>> Sent: Friday, April 5, 2019 2:10 PM
>>>>>>> To: Carissa Klemmer - NOAA Federal
>>>>>>> Cc: Pete Pokrandt; Derek VanPelt - NOAA Affiliate; Gilbert Sebenste;
>>>>>>> conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: Large lags on CONDUIT feed - started a week or so ago
>>>>>>>
>>>>>>> Carissa,
>>>>>>>
>>>>>>> The Boulder connection is definitely performing very well for CONDUIT.
>>>>>>> Although there have been a couple of little blips (~ 120 seconds) since
>>>>>>> yesterday, overall the performance is superb. I don't think it's quite
>>>>>>> as clean as prior to the ~February 10th date when the D.C. connection
>>>>>>> went bad, but it's still excellent performance. Here's our graph now
>>>>>>> with a single connection (no splits):
>>>>>>> <pastedImage.png>
>>>>>>> My next question is: Will CONDUIT stay pointing at Boulder until D.C.
>>>>>>> is fixed, or might you be required to switch back to D.C. at some point
>>>>>>> before that?
>>>>>>>
>>>>>>> Thanks... Art
>>>>>>>
>>>>>>> Arthur A. Person
>>>>>>> Assistant Research Professor, System Administrator
>>>>>>> Penn State Department of Meteorology and Atmospheric Science
>>>>>>> email: aap1@xxxxxxx, phone: 814-863-1563
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>
>>>>>>> Sent: Thursday, April 4, 2019 6:22 PM
>>>>>>> To: Person, Arthur A.
>>>>>>> Cc: Pete Pokrandt; Derek VanPelt - NOAA Affiliate; Gilbert Sebenste;
>>>>>>> conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: Large lags on CONDUIT feed - started a week or so ago
>>>>>>>
>>>>>>> Catching up here.
>>>>>>>
>>>>>>> Derek,
>>>>>>> Do we have traceroutes from all users? Does anything in VCenter show
>>>>>>> any system resource constraints?
>>>>>>>
>>>>>>> On Thursday, April 4, 2019, Person, Arthur A. <aap1@xxxxxxx> wrote:
>>>>>>> Yeh, definitely looks "blipier" starting around 7Z this morning, but
>>>>>>> nothing like it was before. And all last night was clean. Here's our
>>>>>>> graph with a 2-way split, a huge improvement over what it was before
>>>>>>> the switch to Boulder:
>>>>>>>
>>>>>>>
>>>>>>> Agree with Pete that this morning's data probably isn't a good test
>>>>>>> since there were other factors. Since this seems so much better, I'm
>>>>>>> going to try switching to no split as an experiment and see how it
>>>>>>> holds up.
>>>>>>>
>>>>>>> Art
>>>>>>>
>>>>>>> Arthur A. Person
>>>>>>> Assistant Research Professor, System Administrator
>>>>>>> Penn State Department of Meteorology and Atmospheric Science
>>>>>>> email: aap1@xxxxxxx, phone: 814-863-1563
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Pete Pokrandt <poker@xxxxxxxxxxxx>
>>>>>>> Sent: Thursday, April 4, 2019 1:51 PM
>>>>>>> To: Derek VanPelt - NOAA Affiliate
>>>>>>> Cc: Person, Arthur A.; Gilbert Sebenste; Anne Myckow - NOAA Affiliate;
>>>>>>> conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [Ncep.list.pmb-dataflow] [conduit] Large lags on CONDUIT
>>>>>>> feed - started a week or so ago
>>>>>>>
>>>>>>> Ah, so perhaps not a good test.. I'll set it back to a 5-way split and
>>>>>>> see how it looks tomorrow.
>>>>>>>
>>>>>>> Thanks for the info,
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: Derek VanPelt - NOAA Affiliate <derek.vanpelt@xxxxxxxx>
>>>>>>> Sent: Thursday, April 4, 2019 12:38 PM
>>>>>>> To: Pete Pokrandt
>>>>>>> Cc: Person, Arthur A.; Gilbert Sebenste; Anne Myckow - NOAA Affiliate;
>>>>>>> conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [Ncep.list.pmb-dataflow] [conduit] Large lags on CONDUIT
>>>>>>> feed - started a week or so ago
>>>>>>>
>>>>>>> HI Pete -- we did have a separate issu hit the CONDUIT feed today. We
>>>>>>> should be recovering now, but the backlog was sizeable. If these
>>>>>>> numbers are not back to the baseline in the next hour or so please let
>>>>>>> us know. We are also watching our queues and they are decreasing, but
>>>>>>> not as quickly as we had hoped.
>>>>>>>
>>>>>>> Thank you,
>>>>>>>
>>>>>>> Derek
>>>>>>>
>>>>>>> On Thu, Apr 4, 2019 at 1:26 PM 'Pete Pokrandt' via _NCEP
>>>>>>> list.pmb-dataflow <ncep.list.pmb-dataflow@xxxxxxxx> wrote:
>>>>>>> FYI - there is still a much larger lag for the 12 UTC run with a 5-way
>>>>>>> split compared to a 10-way split. It's better since everything else
>>>>>>> failed over to Boulder, but I'd venture to guess that's not the root of
>>>>>>> the problem.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Prior to whatever is going on to cause this, I don'r recall ever seeing
>>>>>>> lags this large with a 5-way split. It looked much more like the left
>>>>>>> hand side of this graph, with small increases in lag with each 6 hourly
>>>>>>> model run cycle, but more like 100 seconds vs the ~900 that I got this
>>>>>>> morning.
>>>>>>>
>>>>>>> FYI I am going to change back to a 10 way split for now.
>>>>>>>
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt
>>>>>>> <poker@xxxxxxxxxxxx>
>>>>>>> Sent: Wednesday, April 3, 2019 4:57 PM
>>>>>>> To: Person, Arthur A.; Gilbert Sebenste; Anne Myckow - NOAA Affiliate
>>>>>>> Cc: conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>>>> feed - started a week or so ago
>>>>>>>
>>>>>>> Sorry, was out this morning and just had a chance to look into this. I
>>>>>>> concur with Art and Gilbert that things appear to have gotten better
>>>>>>> starting with the failover of everything else to Boulder yesterday. I
>>>>>>> will also reconfigure to go back to a 5-way split (as opposed to the
>>>>>>> 10-way split that I've been using since this issue began) and keep an
>>>>>>> eye on tomorrow's 12 UTC model run cycle - if the lags go up, it
>>>>>>> usually happens worst during that cycle, shortly before 18 UTC each
>>>>>>> day.
>>>>>>>
>>>>>>> I'll report back tomorrow how it looks, or you can see at
>>>>>>>
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Person, Arthur A.
>>>>>>> <aap1@xxxxxxx>
>>>>>>> Sent: Wednesday, April 3, 2019 4:04 PM
>>>>>>> To: Gilbert Sebenste; Anne Myckow - NOAA Affiliate
>>>>>>> Cc: conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>>>> feed - started a week or so ago
>>>>>>>
>>>>>>> Anne,
>>>>>>>
>>>>>>> I'll hop back in the loop here... for some reason these replies started
>>>>>>> going into my junk file (bleh). Anyway, I agree with Gilbert's
>>>>>>> assessment. Things turned real clean around 12Z yesterday, looking at
>>>>>>> the graphs. I usually look at flood.atmos.uiuc.edu when there are
>>>>>>> problem as their connection always seems to be the cleanest. If there
>>>>>>> are even small blips or ups and downs in their latencies, that usually
>>>>>>> means there's a network aberration somewhere that usually amplifies
>>>>>>> into hundreds or thousands of seconds at our site and elsewhere.
>>>>>>> Looking at their graph now, you can see the blipiness up until 12Z
>>>>>>> yesterday, and then it's flat (except for the one spike around 16Z
>>>>>>> today which I would ignore):
>>>>>>>
>>>>>>> <pastedImage.png>
>>>>>>> Our direct-connected site, which is using a 10-way split right now,
>>>>>>> also shows a return to calmness in the latencies:
>>>>>>>
>>>>>>> Prior to the recent latency jump, I did not use split requests and the
>>>>>>> reception had been stellar for quite some time. It's my suspicion that
>>>>>>> this is a networking congestion issue somewhere close to the source
>>>>>>> since it seems to affect all downstream sites. For that reason, I
>>>>>>> don't think solving this problem should necessarily involve upgrading
>>>>>>> your server software, but rather identifying what's jamming up the
>>>>>>> network near D.C., and testing this by switching to Boulder was an
>>>>>>> excellent idea. I will now try switching our system to a two-way split
>>>>>>> to see if this performance holds up with fewer pipes. Thanks for your
>>>>>>> help and I'll let you know what I find out.
>>>>>>>
>>>>>>> Art
>>>>>>>
>>>>>>> Arthur A. Person
>>>>>>> Assistant Research Professor, System Administrator
>>>>>>> Penn State Department of Meteorology and Atmospheric Science
>>>>>>> email: aap1@xxxxxxx, phone: 814-863-1563
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Gilbert Sebenste
>>>>>>> <gilbert@xxxxxxxxxxxxxxxx>
>>>>>>> Sent: Wednesday, April 3, 2019 4:07 PM
>>>>>>> To: Anne Myckow - NOAA Affiliate
>>>>>>> Cc: conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] [Ncep.list.pmb-dataflow] Large lags on CONDUIT
>>>>>>> feed - started a week or so ago
>>>>>>>
>>>>>>> Hello Anne,
>>>>>>>
>>>>>>> I'll jump in here as well. Consider the CONDUIT delays at UNIDATA:
>>>>>>>
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
>>>>>>>
>>>>>>>
>>>>>>> And now, Wisconsin:
>>>>>>>
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>>>>>>
>>>>>>> And finally, the University of Washington:
>>>>>>>
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+freshair1.atmos.washington.edu
>>>>>>>
>>>>>>>
>>>>>>> All three of whom have direct feeds from you. Flipping over to Boulder
>>>>>>> definitely caused a major improvement. There was still a brief spike in
>>>>>>> delay, but much shorter and minimal
>>>>>>> compared to what it was.
>>>>>>>
>>>>>>> Gilbert
>>>>>>>
>>>>>>> On Wed, Apr 3, 2019 at 10:03 AM Anne Myckow - NOAA Affiliate
>>>>>>> <anne.myckow@xxxxxxxx> wrote:
>>>>>>> Hi Pete,
>>>>>>>
>>>>>>> As of yesterday we failed almost all of our applications to our site in
>>>>>>> Boulder (meaning away from CONDUIT). Have you noticed an improvement in
>>>>>>> your speeds since yesterday afternoon? If so this will give us a clue
>>>>>>> that maybe there's something interfering on our side that isn't
>>>>>>> specifically CONDUIT, but another app that might be causing congestion.
>>>>>>> (And if it's the same then that's a clue in the other direction.)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Anne
>>>>>>>
>>>>>>> On Mon, Apr 1, 2019 at 3:24 PM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>>>>>> The lag here at UW-Madison was up to 1200 seconds today, and that's
>>>>>>> with a 10-way split feed. Whatever is causing the issue has definitely
>>>>>>> not been resolved, and historically is worse during the work week than
>>>>>>> on the weekends. If that helps at all.
>>>>>>>
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: Anne Myckow - NOAA Affiliate <anne.myckow@xxxxxxxx>
>>>>>>> Sent: Thursday, March 28, 2019 4:28 PM
>>>>>>> To: Person, Arthur A.
>>>>>>> Cc: Carissa Klemmer - NOAA Federal; Pete Pokrandt;
>>>>>>> _NCEP.List.pmb-dataflow; conduit@xxxxxxxxxxxxxxxx;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed -
>>>>>>> started a week or so ago
>>>>>>>
>>>>>>> Hello Art,
>>>>>>>
>>>>>>> We will not be upgrading to version 6.13 on these systems as they are
>>>>>>> not robust enough to support the local logging inherent in the new
>>>>>>> version .
>>>>>>>
>>>>>>> I will check in with my team on if there are any further actions we can
>>>>>>> take to try and troubleshoot this issue, but I fear we may be at the
>>>>>>> limit of our ability to make this better.
>>>>>>>
>>>>>>> I’ll let you know tomorrow where we stand. Thanks.
>>>>>>> Anne
>>>>>>>
>>>>>>> On Mon, Mar 25, 2019 at 3:00 PM Person, Arthur A. <aap1@xxxxxxx> wrote:
>>>>>>> Carissa,
>>>>>>>
>>>>>>> Can you report any status on this inquiry?
>>>>>>>
>>>>>>> Thanks... Art
>>>>>>>
>>>>>>> Arthur A. Person
>>>>>>> Assistant Research Professor, System Administrator
>>>>>>> Penn State Department of Meteorology and Atmospheric Science
>>>>>>> email: aap1@xxxxxxx, phone: 814-863-1563
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>
>>>>>>> Sent: Tuesday, March 12, 2019 8:30 AM
>>>>>>> To: Pete Pokrandt
>>>>>>> Cc: Person, Arthur A.; conduit@xxxxxxxxxxxxxxxx;
>>>>>>> support-conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow
>>>>>>> Subject: Re: Large lags on CONDUIT feed - started a week or so ago
>>>>>>>
>>>>>>> Hi Everyone
>>>>>>>
>>>>>>> I’ve added the Dataflow team email to the thread. I haven’t heard that
>>>>>>> any changes were made or that any issues were found. But the team can
>>>>>>> look today and see if we have any signifiers of overall slowness with
>>>>>>> anything.
>>>>>>>
>>>>>>> Dataflow, try taking a look at the new Citrix or VM troubleshooting
>>>>>>> tools if there are any abnormal signatures that may explain this.
>>>>>>>
>>>>>>> On Monday, March 11, 2019, Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>>>>>>> Art,
>>>>>>>
>>>>>>> I don't know if NCEP ever figured anything out, but I've been able to
>>>>>>> keep my latencies reasonable (300-600s max, mostly during the 12 UTC
>>>>>>> model suite) by splitting my CONDUIT request 10 ways, instead of the 5
>>>>>>> that I had been doing, or in a single request. Maybe give that a try
>>>>>>> and see if it helps at all.
>>>>>>>
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: Person, Arthur A. <aap1@xxxxxxx>
>>>>>>> Sent: Monday, March 11, 2019 3:45 PM
>>>>>>> To: Holly Uhlenhake - NOAA Federal; Pete Pokrandt
>>>>>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or
>>>>>>> so ago
>>>>>>>
>>>>>>> Holly,
>>>>>>>
>>>>>>> Was there any resolution to this on the NCEP end? I'm still seeing
>>>>>>> terrible delays (1000-4000 seconds) receiving data from
>>>>>>> conduit.ncep.noaa.gov. It would be helpful to know if things are
>>>>>>> resolved at NCEP's end so I know whether to look further down the line.
>>>>>>>
>>>>>>> Thanks... Art
>>>>>>>
>>>>>>> Arthur A. Person
>>>>>>> Assistant Research Professor, System Administrator
>>>>>>> Penn State Department of Meteorology and Atmospheric Science
>>>>>>> email: aap1@xxxxxxx, phone: 814-863-1563
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Holly Uhlenhake - NOAA
>>>>>>> Federal <holly.uhlenhake@xxxxxxxx>
>>>>>>> Sent: Thursday, February 21, 2019 12:05 PM
>>>>>>> To: Pete Pokrandt
>>>>>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or
>>>>>>> so ago
>>>>>>>
>>>>>>> Hi Pete,
>>>>>>>
>>>>>>> We'll take a look and see if we can figure out what might be going on.
>>>>>>> We haven't done anything to try and address this yet, but based on your
>>>>>>> analysis I'm suspicious that it might be tied to a resource constraint
>>>>>>> on the VM or the blade it resides on.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Holly Uhlenhake
>>>>>>> Acting Dataflow Team Lead
>>>>>>>
>>>>>>> On Thu, Feb 21, 2019 at 11:32 AM Pete Pokrandt <poker@xxxxxxxxxxxx>
>>>>>>> wrote:
>>>>>>> Just FYI, data is flowing, but the large lags continue.
>>>>>>>
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
>>>>>>>
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt
>>>>>>> <poker@xxxxxxxxxxxx>
>>>>>>> Sent: Wednesday, February 20, 2019 12:07 PM
>>>>>>> To: Carissa Klemmer - NOAA Federal
>>>>>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or
>>>>>>> so ago
>>>>>>>
>>>>>>> Data is flowing again - picked up somewhere in the GEFS. Maybe CONDUIT
>>>>>>> server was restarted, or ldm on it? Lags are large (3000s+) but
>>>>>>> dropping slowly
>>>>>>>
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt
>>>>>>> <poker@xxxxxxxxxxxx>
>>>>>>> Sent: Wednesday, February 20, 2019 11:56 AM
>>>>>>> To: Carissa Klemmer - NOAA Federal
>>>>>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: Re: [conduit] Large lags on CONDUIT feed - started a week or
>>>>>>> so ago
>>>>>>>
>>>>>>> Just a quick follow-up - we started falling far enough behind (3600+
>>>>>>> sec) that we are losing data. We got short files starting at 174h into
>>>>>>> the GFS run, and only got (incomplete) data through 207h.
>>>>>>>
>>>>>>> We have now not received any data on CONDUIT since 11:27 AM CST (1727
>>>>>>> UTC) today (Wed Feb 20)
>>>>>>>
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>>
>>>>>>>
>>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx
>>>>>>> <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt
>>>>>>> <poker@xxxxxxxxxxxx>
>>>>>>> Sent: Wednesday, February 20, 2019 11:28 AM
>>>>>>> To: Carissa Klemmer - NOAA Federal
>>>>>>> Cc: conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
>>>>>>> Subject: [conduit] Large lags on CONDUIT feed - started a week or so ago
>>>>>>>
>>>>>>> Carissa,
>>>>>>>
>>>>>>> We have been feeding CONDUIT using a 5 way split feed direct from
>>>>>>> conduit.ncep.noaa.gov, and it had been really good for some time, lags
>>>>>>> 30-60 seconds or less.
>>>>>>>
>>>>>>> However, the past week or so, we've been seeing some very large lags
>>>>>>> during each 6 hour model suite - Unidata is also seeing these - they
>>>>>>> are also feeding direct from conduit.ncep.noaa.gov.
>>>>>>>
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
>>>>>>>
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
>>>>>>>
>>>>>>>
>>>>>>> Any idea what's going on, or how we can find out?
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Pete
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>>>> _______________________________________________
>>>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>>>>> available through the web. Users who post to any of the lists we
>>>>>>> maintain are reminded to remove any personal information that they
>>>>>>> do not want to be made public.
>>>>>>>
>>>>>>>
>>>>>>> conduit mailing list
>>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>>> For list information or to unsubscribe, visit:
>>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Carissa Klemmer
>>>>>>> NCEP Central Operations
>>>>>>> IDSB Branch Chief
>>>>>>> 301-683-3835
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Ncep.list.pmb-dataflow mailing list
>>>>>>> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>>>>>>> --
>>>>>>> Anne Myckow
>>>>>>> Lead Dataflow Analyst
>>>>>>> NOAA/NCEP/NCO
>>>>>>> 301-683-3825
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Anne Myckow
>>>>>>> Lead Dataflow Analyst
>>>>>>> NOAA/NCEP/NCO
>>>>>>> 301-683-3825
>>>>>>> _______________________________________________
>>>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>>>>> available through the web. Users who post to any of the lists we
>>>>>>> maintain are reminded to remove any personal information that they
>>>>>>> do not want to be made public.
>>>>>>>
>>>>>>>
>>>>>>> conduit mailing list
>>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>>> For list information or to unsubscribe, visit:
>>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ----
>>>>>>>
>>>>>>> Gilbert Sebenste
>>>>>>> Consulting Meteorologist
>>>>>>> AllisonHouse, LLC
>>>>>>> _______________________________________________
>>>>>>> Ncep.list.pmb-dataflow mailing list
>>>>>>> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Derek Van Pelt
>>>>>>> DataFlow Analyst
>>>>>>> NOAA/NCEP/NCO
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Carissa Klemmer
>>>>>>> NCEP Central Operations
>>>>>>> IDSB Branch Chief
>>>>>>> 301-683-3835
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>>>>> available through the web. Users who post to any of the lists we
>>>>>>> maintain are reminded to remove any personal information that they
>>>>>>> do not want to be made public.
>>>>>>>
>>>>>>>
>>>>>>> conduit mailing list
>>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>>> For list information or to unsubscribe, visit:
>>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Derek Van Pelt
>>>>>>> DataFlow Analyst
>>>>>>> NOAA/NCEP/NCO
>>>>>>> --
>>>>>>> Misspelled straight from Derek's phone.
>>>>>>> _______________________________________________
>>>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>>>>> available through the web. Users who post to any of the lists we
>>>>>>> maintain are reminded to remove any personal information that they
>>>>>>> do not want to be made public.
>>>>>>>
>>>>>>>
>>>>>>> conduit mailing list
>>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>>> For list information or to unsubscribe, visit:
>>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>> _______________________________________________
>>>>>>> Ncep.list.pmb-dataflow mailing list
>>>>>>> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dustin Sheffler
>>>>>>> NCEP Central Operations - Dataflow
>>>>>>> 5830 University Research Court, Rm 1030
>>>>>>> College Park, Maryland 20740
>>>>>>> Office: (301) 683-3827
>>>>>>> _______________________________________________
>>>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>>>>> available through the web. Users who post to any of the lists we
>>>>>>> maintain are reminded to remove any personal information that they
>>>>>>> do not want to be made public.
>>>>>>>
>>>>>>>
>>>>>>> conduit mailing list
>>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>>> For list information or to unsubscribe, visit:
>>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>
>>>>>
>>>>> --
>>>>> Derek Van Pelt
>>>>> DataFlow Analyst
>>>>> NOAA/NCEP/NCO
>>>>> _______________________________________________
>>>>> Ncep.list.pmb-dataflow mailing list
>>>>> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>>>>
>>>>
>>>> --
>>>> Anne Myckow
>>>> Lead Dataflow Analyst
>>>> NOAA/NCEP/NCO
>>>> 301-683-3825
>>
>>
>> --
>> Anne Myckow
>> Lead Dataflow Analyst
>> NOAA/NCEP/NCO
>> 301-683-3825
>
>
> --
> Anne Myckow
> Lead Dataflow Analyst
> NOAA/NCEP/NCO
> 301-683-3825
