Re: [conduit] [Ncep.list.pmb-dataflow] How's your GFS?

  • To: Michael Shedlock <michael.shedlock@xxxxxxxx>
  • Subject: Re: [conduit] [Ncep.list.pmb-dataflow] How's your GFS?
  • From: Arthur A Person <aap1@xxxxxxx>
  • Date: Fri, 13 Nov 2015 15:27:33 -0500 (EST)
Mike, 

Here's our traceroute from iddrs2a.meteo.psu.edu: 

traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte 
packets 
1 192.5.158.1 (192.5.158.1) 0.324 ms 0.261 ms 172.29.0.66 (172.29.0.66) 0.213 
ms 
2 White1-ethernet3-1.gw.psu.edu (172.30.5.177) 0.282 ms 0.323 ms 0.297 ms 
3 Windstream1-ethernet2-1.gw.psu.edu (172.30.5.106) 0.385 ms 0.404 ms 
Telecom5-ethernet3-1.gw.psu.edu (172.30.5.110) 0.379 ms 
4 Telecom5-ethernet2-1.gw.psu.edu (172.30.8.10) 0.550 ms 0.453 ms 
et-8-0-0.2364.rtr.chic.net.internet2.edu (198.71.45.156) 15.245 ms 
5 et-10-0-0.107.rtr.clev.net.internet2.edu (198.71.45.9) 25.017 ms 
et-8-0-0.2364.rtr.chic.net.internet2.edu (198.71.45.156) 15.275 ms 15.410 ms 
6 et-11-3-0-1276.clpk-core.maxgigapop.net (206.196.177.4) 34.400 ms 34.605 ms 
34.570 ms 
7 noaa-i2.demarc.maxgigapop.net (206.196.177.118) 34.960 ms 35.106 ms 35.244 ms 
8 noaa-i2.demarc.maxgigapop.net (206.196.177.118) 35.558 ms 35.376 ms 
140.90.111.36 (140.90.111.36) 34.475 ms 
9 140.90.111.36 (140.90.111.36) 34.819 ms 34.776 ms 140.90.76.69 (140.90.76.69) 
34.935 ms 
10 140.90.76.69 (140.90.76.69) 34.838 ms 34.905 ms 34.987 ms 
11 * * * 
12 * * * 

Art 

> From: "Michael Shedlock" <michael.shedlock@xxxxxxxx>
> To: "Mike Dross" <mdross@xxxxxxxxxxxxxxxxxx>, "Arthur A Person" <aap1@xxxxxxx>
> Cc: "Bentley, Alicia M" <ambentley@xxxxxxxxxx>, "Michael Schmidt"
> <mschmidt@xxxxxxxx>, "support-conduit@xxxxxxxxxxxxxxxx"
> <conduit@xxxxxxxxxxxxxxxx>, "_NCEP.List.pmb-dataflow"
> <ncep.list.pmb-dataflow@xxxxxxxx>, "Carissa Klemmer - NOAA Federal"
> <carissa.l.klemmer@xxxxxxxx>, "Daes Support" <daessupport@xxxxxxxxxx>
> Sent: Friday, November 13, 2015 2:53:56 PM
> Subject: Re: [Ncep.list.pmb-dataflow] [conduit] How's your GFS?

> All,

> NCEP is indeed on internet2, which I presume would apply here.

> A couple of noteworthy things.... I see some latency, but not for everyone, 
> and
> it doesn't seem to matter which conduit machine a client is connected to. For
> example, with today's and yesterday's gfs.t12z.pgrb2.0p25.f096 (hour 96) file
> here are the latencies per client that I see:

>> 11/12
>> Wisconsin: A few seconds
>> Unidata/UCAR: A few seconds
>> UIUC: 13 minutes
>> PSU: 27 minutes

>> 11/13:
>> Wisconsin: A few seconds
>> Unidata/UCAR: A few seconds
>> UIUC: 2.33 minutes
>> PSU: 2.75 minutes

> Another correlation is that UIUC and PSU (the ones with latency) are only 
> using
> one thread to connect to our conduit, whereas Wisc. and Unidata use multiple
> threads.

> At the moment this sort of has the appearance of a bottleneck outside of NCEP.
> It might also be useful to see traceroutes from UIUC and PSU to NCEP's 
> CONDUIT.
> I know I saw some traceroutes below. Can you try that and share with us?

> Mike Shedlock
> NCEP Central Operations
> Dataflow Team
> 301.683.3834

> On 11/13/2015 11:42 AM, Mike Dross wrote:

>> My $ 0.02 from having works with LDM since the mid 90's.

>> I assume NCEP is not on internet2? If so bandwidth shouldn't be an issue .
>> Regardless I would check the traceroutes to ensure a good path, high 
>> bandwidth,
>> low latency. Basic network topology check. I am sure you have done this.

>> An iperf test is a simple way to test the maximum throughput to see if 
>> bandwidth
>> is an issue. If that's not it, high latency or the way LDM is set up on the
>> upstream side is likely the culprit.

>> Mike

>> Sent from my iPad

>> On Nov 13, 2015, at 10:05 AM, Arthur A Person < aap1@xxxxxxx > wrote:

>>> Carissa,

>>> Yes, still issues. There was a period several weeks ago when throughput was
>>> clean, but recently we've seen delays to varying degrees.
>>> Based on the Unidata latency chart from our reported statistics (
>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+iddrs2a.meteo.psu.edu
>>> ),
>>> we've seen delays during 0.25 degree gfs transmission that range from 500
>>> seconds to 3500 seconds over the past couple of days.
>>> Also, comparison with charts from other schools seem to show better 
>>> reception
>>> when feeding from "conduit1" rather than "conduit2".
>>> Does this mean anything to you or is it purely coincidence or incidental?

>>> Thanks for any insights you can provide.

>>> Art

>>>> From: "Carissa Klemmer - NOAA Federal" < carissa.l.klemmer@xxxxxxxx >
>>>> To: "Arthur A Person" < aap1@xxxxxxx >, "_NCEP.List.pmb-dataflow" <
>>>> ncep.list.pmb-dataflow@xxxxxxxx >
>>>> Cc: " support-conduit@xxxxxxxxxxxxxxxx " < conduit@xxxxxxxxxxxxxxxx >, 
>>>> "Pete
>>>> Pokrandt" < poker@xxxxxxxxxxxx >, "Michael Schmidt" < mschmidt@xxxxxxxx >,
>>>> "Bentley, Alicia M" < ambentley@xxxxxxxxxx >, "Daes Support" <
>>>> daessupport@xxxxxxxxxx >
>>>> Sent: Friday, November 13, 2015 9:26:28 AM
>>>> Subject: Re: [conduit] How's your GFS?

>>>> Art,
>>>> I am going to add our team to this thread. Are you still seeing issues. Is 
>>>> so we
>>>> will take a look and see if we can tell if anything on our side is 
>>>> happening
>>>> around FH 96.

>>>> Carissa Klemmer
>>>> NCEP Central Operations
>>>> Dataflow Team Lead
>>>> 301-683-3835

>>>> On Thu, Nov 5, 2015 at 4:23 PM, Arthur A Person < aap1@xxxxxxx > wrote:

>>>>> Hi all...

>>>>> Conduit latencies have crept upward again for the past few weeks... not
>>>>> unbearable, but still significant.
>>>>> At first it seemed to only affect us, but it looks like it's affecting 
>>>>> UIUC now
>>>>> also, but not so much Wisconsin.
>>>>> Inspecting our logs, we've noticed that there's no delay out to about 90 
>>>>> hours
>>>>> of gfs transmission, but
>>>>> starting at 96 hours, the delays start to ramp up steadily. I'm not sure 
>>>>> how to
>>>>> explain that unless something
>>>>> else starts transmitting during that time that competes for bandwidth. 
>>>>> Also, I
>>>>> notice that sites receiving data
>>>>> from "conduit1" seem to be faring better than "conduit2". Is there any
>>>>> difference between these two
>>>>> originating systems or is that just coincidental? Anyone have anything 
>>>>> new to
>>>>> report on this issue?

>>>>> Thanks... Art

>>>>>> From: "Pete Pokrandt" < poker@xxxxxxxxxxxx >
>>>>>> To: "Carissa Klemmer - NOAA Federal" < carissa.l.klemmer@xxxxxxxx >,
>>>>>> mschmidt@xxxxxxxx
>>>>>> Cc: "Bentley, Alicia M" < ambentley@xxxxxxxxxx >, "Daes Support" <
>>>>>> daessupport@xxxxxxxxxx >, " support-conduit@xxxxxxxxxxxxxxxx " <
>>>>>> conduit@xxxxxxxxxxxxxxxx >
>>>>>> Sent: Thursday, September 24, 2015 1:29:59 PM

>>>>>> Subject: Re: [conduit] How's your GFS?

>>>>>> Here are traceroutes from idd.aos.wisc.edu to conduit.ncep.noaa.gov and
>>>>>> ncepldm4.woc.noaa.gov taken at 17:17 UTC, right in the middle of the 18 
>>>>>> UTC GFS
>>>>>> lag spike today.

>>>>>> [ldm@idd ~/etc]$ traceroute conduit.ncep.noaa.gov
>>>>>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte
>>>>>> packets
>>>>>> 1 r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3) 0.833 
>>>>>> ms
>>>>>> 0.819 ms 0.855 ms
>>>>>> 2 internet2-ord-600w-100G.net.wisc.edu (144.92.254.229) 18.077 ms 18.095 
>>>>>> ms
>>>>>> 18.067 ms
>>>>>> 3 et-7-0-0.115.rtr.wash.net.internet2.edu (198.71.45.57) 35.125 ms 
>>>>>> 35.278 ms
>>>>>> 35.261 ms
>>>>>> 4 198.71.45.228 (198.71.45.228) 35.378 ms 35.368 ms 35.335 ms
>>>>>> 5 ae0.clpk-core.maxgigapop.net (206.196.178.81) 36.401 ms 36.408 ms 
>>>>>> 36.284 ms
>>>>>> 6 noaa-rtr.maxgigapop.net (206.196.177.118) 36.523 ms 36.640 ms 36.411 ms
>>>>>> 7 140.90.111.36 (140.90.111.36) 68.769 ms 52.236 ms 52.210 ms
>>>>>> 8 140.90.76.69 (140.90.76.69) 36.602 ms 36.503 ms 36.827 ms
>>>>>> 9 * * *
>>>>>> 10 * * *
>>>>>> ...

>>>>>> [ldm@idd ~/etc]$ traceroute ncepldm4.woc.noaa.gov
>>>>>> traceroute to ncepldm4.woc.noaa.gov (140.172.17.205), 30 hops max, 60 
>>>>>> byte
>>>>>> packets
>>>>>> 1 r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3) 0.838 
>>>>>> ms
>>>>>> 0.847 ms 0.822 ms
>>>>>> 2 internet2-ord-600w-100G.net.wisc.edu (144.92.254.229) 18.513 ms 18.506 
>>>>>> ms
>>>>>> 18.484 ms
>>>>>> 3 ae0.3454.core-l3.frgp.net (192.43.217.223) 40.245 ms 40.204 ms 40.123 
>>>>>> ms
>>>>>> 4 noaa-i2.frgp.net (128.117.243.11) 43.617 ms 43.544 ms 43.699 ms
>>>>>> 5 2001-mlx8-eth-1-2.boulder.noaa.gov (140.172.2.18) 40.960 ms 40.951 ms 
>>>>>> 41.058
>>>>>> ms
>>>>>> 6 mdf-rtr-6.boulder.noaa.gov (140.172.6.251) 46.516 ms 40.962 ms 40.876 
>>>>>> ms
>>>>>> 7 * * *
>>>>>> 8 * * *
>>>>>> ...

>>>>>> --
>>>>>> Pete Pokrandt - Systems Programmer
>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx

>>>>>> From: conduit-bounces@xxxxxxxxxxxxxxxx < 
>>>>>> conduit-bounces@xxxxxxxxxxxxxxxx > on
>>>>>> behalf of Carissa Klemmer - NOAA Federal < carissa.l.klemmer@xxxxxxxx >
>>>>>> Sent: Thursday, September 24, 2015 10:36 AM
>>>>>> To: mschmidt@xxxxxxxx
>>>>>> Cc: Bentley, Alicia M; Daes Support; support-conduit@xxxxxxxxxxxxxxxx
>>>>>> Subject: Re: [conduit] How's your GFS?
>>>>>> Mike,

>>>>>> Can you provide what server you are coming from. I know your range, but 
>>>>>> I need
>>>>>> to provide to the helpdesk what is primary right now so they can trace 
>>>>>> back.

>>>>>> Carissa Klemmer
>>>>>> NCEP Central Operations
>>>>>> Dataflow Team Lead
>>>>>> 301-683-3835

>>>>>> On Thu, Sep 24, 2015 at 9:57 AM, Mike Schmidt < mschmidt@xxxxxxxx > 
>>>>>> wrote:

>>>>>>> Hi Carissa,

>>>>>>> We've seen the same jump in latencies;

>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+daffy.unidata.ucar.edu

>>>>>>> Here's our traceroute;

>>>>>>> # traceroute conduit.ncep.noaa.gov
>>>>>>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 
>>>>>>> byte
>>>>>>> packets
>>>>>>> 1 flra-n156.unidata.ucar.edu (128.117.156.253) 0.352 ms 0.344 ms 0.325 
>>>>>>> ms
>>>>>>> 2 tcom-gs-1-n243-80.ucar.edu (128.117.243.85) 0.558 ms 0.584 ms 0.662 ms
>>>>>>> 3 xe-0-1-2.873.core-l3.frgp.net (128.117.243.9) 1.138 ms 1.126 ms 1.107 
>>>>>>> ms
>>>>>>> 4 v3454.rtr-chic.frgp.net (192.43.217.222) 23.227 ms 23.296 ms 23.278 ms
>>>>>>> 5 et-7-0-0.115.rtr.wash.net.internet2.edu (198.71.45.57) 40.421 ms 
>>>>>>> 40.408 ms
>>>>>>> 40.340 ms
>>>>>>> 6 198.71.45.228 (198.71.45.228) 40.488 ms 40.649 ms 40.624 ms
>>>>>>> 7 ae0.clpk-core.maxgigapop.net (206.196.178.81) 41.545 ms 41.602 ms 
>>>>>>> 41.170 ms
>>>>>>> 8 noaa-rtr.maxgigapop.net (206.196.177.118) 41.796 ms 41.507 ms 41.592 
>>>>>>> ms
>>>>>>> 9 140.90.111.36 (140.90.111.36) 41.419 ms 41.496 ms 41.623 ms
>>>>>>> 10 140.90.76.69 (140.90.76.69) 41.900 ms 41.728 ms 41.956 ms

>>>>>>> mike

>>>>>>> On Thu, Sep 24, 2015 at 7:49 AM, Carissa Klemmer - NOAA Federal <
>>>>>>> carissa.l.klemmer@xxxxxxxx > wrote:

>>>>>>>> Hi all,

>>>>>>>> I have opened a ticket with our helpdesk and included PSU traceroute. 
>>>>>>>> But can I
>>>>>>>> get a better handle on all the paths that are seeing latencies to
>>>>>>>> conduit.ncep.noaa.gov ? Is both PSU and WISC seeing spikes? Can I get 
>>>>>>>> a WISC
>>>>>>>> traceroute also please?

>>>>>>>> Thanks,

>>>>>>>> Carissa Klemmer
>>>>>>>> NCEP Central Operations
>>>>>>>> Dataflow Team Lead
>>>>>>>> 301-683-3835

>>>>>>>> On Thu, Sep 24, 2015 at 8:19 AM, Arthur A Person < aap1@xxxxxxx > 
>>>>>>>> wrote:

>>>>>>>>> Pete,

>>>>>>>>> I was thinking that too! If I only hadn't sent that email... :)

>>>>>>>>> Anyway, the delays aren't as bad as they were (at least here), but 
>>>>>>>>> are still
>>>>>>>>> indicative of a
>>>>>>>>> lurking problem. Almost seems as though some packet shaping is going 
>>>>>>>>> on, as Tom
>>>>>>>>> suggested previously. Maybe paths get overloaded and something kicks 
>>>>>>>>> in and
>>>>>>>>> meters-out
>>>>>>>>> usage??? Just speculating. I've asked our network folks here to see 
>>>>>>>>> if they can
>>>>>>>>> investigate
>>>>>>>>> our path to NCEP, but that may take awhile. Our traceroute from this 
>>>>>>>>> morning at
>>>>>>>>> 1113Z is:

>>>>>>>>> [ldm@iddrs1a ~]$ traceroute conduit.ncep.noaa.gov
>>>>>>>>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 
>>>>>>>>> byte
>>>>>>>>> packets
>>>>>>>>> 1 172.29.0.66 (172.29.0.66) 0.882 ms 192.5.158.1 (192.5.158.1) 0.278 
>>>>>>>>> ms 0.264 ms
>>>>>>>>> 2 Blue1-ethernet3-1.gw.psu.edu (172.30.5.178) 0.220 ms
>>>>>>>>> White1-ethernet3-1.gw.psu.edu (172.30.5.177) 0.530 ms 0.526 ms
>>>>>>>>> 3 Windstream1-ethernet2-1.gw.psu.edu (172.30.5.106) 0.385 ms
>>>>>>>>> Telecom5-ethernet2-2.gw.psu.edu (172.30.5.102) 0.370 ms
>>>>>>>>> Windstream1-ethernet3-2.gw.psu.edu (172.30.5.114) 0.391 ms
>>>>>>>>> 4 Telecom5-ethernet2-1.gw.psu.edu (172.30.8.10) 0.391 ms 0.408 ms
>>>>>>>>> et-8-0-0.2364.rtr.chic.net.internet2.edu (64.57.30.2) 15.149 ms
>>>>>>>>> 5 et-7-0-0.115.rtr.wash.net.internet2.edu (198.71.45.57) 32.276 ms
>>>>>>>>> et-8-0-0.2364.rtr.chic.net.internet2.edu (64.57.30.2) 15.301 ms
>>>>>>>>> et-7-0-0.115.rtr.wash.net.internet2.edu (198.71.45.57) 32.594 ms
>>>>>>>>> 6 198.71.45.228 (198.71.45.228) 32.423 ms
>>>>>>>>> et-7-0-0.115.rtr.wash.net.internet2.edu (198.71.45.57) 32.431 ms 
>>>>>>>>> 198.71.45.228
>>>>>>>>> (198.71.45.228) 32.843 ms
>>>>>>>>> 7 198.71.45.228 (198.71.45.228) 32.853 ms ae0.clpk-core.maxgigapop.net
>>>>>>>>> (206.196.178.81) 33.407 ms 33.401 ms
>>>>>>>>> 8 ae0.clpk-core.maxgigapop.net (206.196.178.81) 33.858 ms
>>>>>>>>> noaa-rtr.maxgigapop.net (206.196.177.118) 33.483 ms
>>>>>>>>> ae0.clpk-core.maxgigapop.net (206.196.178.81) 33.515 ms
>>>>>>>>> 9 140.90.111.36 (140.90.111.36) 33.574 ms 33.545 ms 
>>>>>>>>> noaa-rtr.maxgigapop.net
>>>>>>>>> (206.196.177.118) 33.907 ms
>>>>>>>>> 10 140.90.76.69 (140.90.76.69) 34.220 ms 34.012 ms 33.901 ms

>>>>>>>>> The above was taken while we were running about 1000 seconds behind. 
>>>>>>>>> A recent
>>>>>>>>> change here
>>>>>>>>> (9/16/2015) was to direct our first hop to Chicago instead of 
>>>>>>>>> Pittsburgh (3rox)
>>>>>>>>> which is now a 100 Gbit
>>>>>>>>> link. Tests to UCAR at that time were showing 1.38 Gbps throughput 
>>>>>>>>> with reduced
>>>>>>>>> latencies. Since
>>>>>>>>> our delays now are not as bad as they were previously, perhaps this 
>>>>>>>>> has helped.
>>>>>>>>> However, there may
>>>>>>>>> still be a choke point further down the line at maxgigapop or 
>>>>>>>>> internal to NCEP
>>>>>>>>> itself. perfsonar monitoring
>>>>>>>>> to NCEP would be useful... does anyone have any tests to NCEP 
>>>>>>>>> currently running?
>>>>>>>>> Can we identify
>>>>>>>>> any common path segments in the trace route above?

>>>>>>>>> Art

>>>>>>>>>> From: "Pete Pokrandt" < poker@xxxxxxxxxxxx >
>>>>>>>>>> To: "Arthur A Person" < aap1@xxxxxxx >, " 
>>>>>>>>>> support-conduit@xxxxxxxxxxxxxxxx " <
>>>>>>>>>> conduit@xxxxxxxxxxxxxxxx >
>>>>>>>>>> Cc: "Bentley, Alicia M" < ambentley@xxxxxxxxxx >, "Daes Support" <
>>>>>>>>>> daessupport@xxxxxxxxxx >, "Carissa Klemmer - NOAA Federal" <
>>>>>>>>>> carissa.l.klemmer@xxxxxxxx >, "Tyle, Kevin R" < ktyle@xxxxxxxxxx >
>>>>>>>>>> Sent: Thursday, September 24, 2015 1:25:31 AM

>>>>>>>>>> Subject: Re: [conduit] How's your GFS?

>>>>>>>>>> Art,

>>>>>>>>>> Looks like you spoke too soon. Big lags ~1000 secs started up again 
>>>>>>>>>> with today's
>>>>>>>>>> 12 UTC cycle. Very mysterious..

>>>>>>>>>> They are showing up on our feed and consequently downstream from us 
>>>>>>>>>> at Albny.

>>>>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu

>>>>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+install.atmos.albany.edu

>>>>>>>>>> <sigh..>

>>>>>>>>>> Pete

>>>>>>>>>> --
>>>>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx

>>>>>>>>>> From: Arthur A Person < aap1@xxxxxxx >
>>>>>>>>>> Sent: Monday, September 21, 2015 10:14 AM
>>>>>>>>>> To: support-conduit@xxxxxxxxxxxxxxxx
>>>>>>>>>> Cc: Bentley, Alicia M; Daes Support; Carissa Klemmer - NOAA Federal; 
>>>>>>>>>> Pete
>>>>>>>>>> Pokrandt; Tyle, Kevin R
>>>>>>>>>> Subject: Re: [conduit] How's your GFS?
>>>>>>>>>> Folks,

>>>>>>>>>> Looks like something changed late on Friday in the network paths 
>>>>>>>>>> affecting Penn
>>>>>>>>>> State and the other universities feeding
>>>>>>>>>> CONDUIT from NCEP... delays have dropped to a crisp 30 seconds or 
>>>>>>>>>> less. Does
>>>>>>>>>> anyone know if a problem was found/fixed?
>>>>>>>>>> I know some issues were addressed at Penn State with some issues 
>>>>>>>>>> still being
>>>>>>>>>> worked-on. Back in the first week of
>>>>>>>>>> September the feeds were good and then degraded... just want to make 
>>>>>>>>>> sure that
>>>>>>>>>> doesn't happen again before I re-enable
>>>>>>>>>> ingest of the gfs 0.25 degree data.

>>>>>>>>>> Art

>>>>>>>>>>> From: "Arthur A Person" < aap1@xxxxxxx >
>>>>>>>>>>> To: " support-conduit@xxxxxxxxxxxxxxxx " < conduit@xxxxxxxxxxxxxxxx 
>>>>>>>>>>> >
>>>>>>>>>>> Cc: "Bentley, Alicia M" < ambentley@xxxxxxxxxx >, "Daes Support" <
>>>>>>>>>>> daessupport@xxxxxxxxxx >, "Carissa Klemmer - NOAA Federal" <
>>>>>>>>>>> carissa.l.klemmer@xxxxxxxx >
>>>>>>>>>>> Sent: Wednesday, September 9, 2015 4:27:26 PM
>>>>>>>>>>> Subject: Re: [conduit] How's your GFS?

>>>>>>>>>>> Just a heads up...

>>>>>>>>>>> I've reconfigured our IDD relay to distribute conduit without the 
>>>>>>>>>>> gfs 0.25
>>>>>>>>>>> degree data until we get our latencies under control. We've got 
>>>>>>>>>>> some issues
>>>>>>>>>>> internal to Penn State creating problems on top of any external 
>>>>>>>>>>> issues and our
>>>>>>>>>>> conduit feed is useless the way it is at the moment. By reverting 
>>>>>>>>>>> to conduit
>>>>>>>>>>> without gfs 0.25, hopefully we'll maintain a useful stream. As soon 
>>>>>>>>>>> as the
>>>>>>>>>>> latencies are addressed, I will reintroduce the gfs 0.25.

>>>>>>>>>>> Art

>>>>>>>>>>>> From: "Arthur A Person" < aap1@xxxxxxx >
>>>>>>>>>>>> To: "Carissa Klemmer - NOAA Federal" < carissa.l.klemmer@xxxxxxxx >
>>>>>>>>>>>> Cc: "Bentley, Alicia M" < ambentley@xxxxxxxxxx >, "Daes Support" <
>>>>>>>>>>>> daessupport@xxxxxxxxxx >, " support-conduit@xxxxxxxxxxxxxxxx " <
>>>>>>>>>>>> conduit@xxxxxxxxxxxxxxxx >
>>>>>>>>>>>> Sent: Wednesday, September 9, 2015 8:00:19 AM
>>>>>>>>>>>> Subject: Re: [conduit] How's your GFS?

>>>>>>>>>>>> All,

>>>>>>>>>>>>> From: "Carissa Klemmer - NOAA Federal" < 
>>>>>>>>>>>>> carissa.l.klemmer@xxxxxxxx >
>>>>>>>>>>>>> To: "Arthur A Person" < aap1@xxxxxxx >
>>>>>>>>>>>>> Cc: "Tyle, Kevin R" < ktyle@xxxxxxxxxx >, "Bentley, Alicia M" <
>>>>>>>>>>>>> ambentley@xxxxxxxxxx >, "Daes Support" < daessupport@xxxxxxxxxx 
>>>>>>>>>>>>> >, "
>>>>>>>>>>>>> support-conduit@xxxxxxxxxxxxxxxx " < conduit@xxxxxxxxxxxxxxxx >
>>>>>>>>>>>>> Sent: Tuesday, September 8, 2015 10:19:06 PM
>>>>>>>>>>>>> Subject: Re: [conduit] How's your GFS?

>>>>>>>>>>>>> All,

>>>>>>>>>>>>> NCEP is not making any active changes to our networks that should 
>>>>>>>>>>>>> affect your
>>>>>>>>>>>>> latencies, especially not over a weekend. I am not aware of any 
>>>>>>>>>>>>> changes that
>>>>>>>>>>>>> occurred over the holiday that would have impacted these 
>>>>>>>>>>>>> networks. This is
>>>>>>>>>>>>> likely downstream of NCEP control which is why you see the 
>>>>>>>>>>>>> latencies come and
>>>>>>>>>>>>> go.

>>>>>>>>>>>> Okay... I guess my interpretation was wrong, then. My apologies. 
>>>>>>>>>>>> There does seem
>>>>>>>>>>>> to be a problem pretty close to NCEP, however, since the latencies 
>>>>>>>>>>>> seem to come
>>>>>>>>>>>> and go at all top-tier sites... although not all sites are the 
>>>>>>>>>>>> same (ours seems
>>>>>>>>>>>> to be the highest). Maybe we're pushing the long-haul connectivity 
>>>>>>>>>>>> to the limit
>>>>>>>>>>>> and multiple choke points are showing up? Time to get our 
>>>>>>>>>>>> networking folks more
>>>>>>>>>>>> involved...

>>>>>>>>>>>> Art

>>>>>>>>>>>>> Carissa Klemmer
>>>>>>>>>>>>> NCEP Central Operations
>>>>>>>>>>>>> Production Management Branch Dataflow Team
>>>>>>>>>>>>> 301-683-3835

>>>>>>>>>>>>> On Tue, Sep 8, 2015 at 1:21 PM, Arthur A Person < aap1@xxxxxxx > 
>>>>>>>>>>>>> wrote:

>>>>>>>>>>>>>> We appear to have had gfs reception problems with 0Z and 6Z runs 
>>>>>>>>>>>>>> last night.
>>>>>>>>>>>>>> After implementation
>>>>>>>>>>>>>> of the 0.25 degree gfs, CONDUIT latencies were very large across 
>>>>>>>>>>>>>> all sites
>>>>>>>>>>>>>> during 0.25 degree data
>>>>>>>>>>>>>> transmission, but a week-or-so ago dropped to negligible levels. 
>>>>>>>>>>>>>> Over the
>>>>>>>>>>>>>> weekend they jumped
>>>>>>>>>>>>>> back up again. I interpret this to mean NCEP is tinkering with 
>>>>>>>>>>>>>> network paths
>>>>>>>>>>>>>> trying to find an effective
>>>>>>>>>>>>>> way to get these huge bursts of data out to the downstream 
>>>>>>>>>>>>>> sites. The gfs data
>>>>>>>>>>>>>> loss last night may have
>>>>>>>>>>>>>> been f rom the large latencies or from other unrelated delivery 
>>>>>>>>>>>>>> problems...
>>>>>>>>>>>>>> dunno...

>>>>>>>>>>>>>> Art

>>>>>>>>>>>>>>> From: "Tyle, Kevin R" < ktyle@xxxxxxxxxx >
>>>>>>>>>>>>>>> To: "Pete Pokrandt" < poker@xxxxxxxxxxxx >
>>>>>>>>>>>>>>> Cc: "Bentley, Alicia M" < ambentley@xxxxxxxxxx >, "Daes 
>>>>>>>>>>>>>>> Support" <
>>>>>>>>>>>>>>> daessupport@xxxxxxxxxx >, conduit@xxxxxxxxxxxxxxxx
>>>>>>>>>>>>>>> Sent: Tuesday, September 8, 2015 1:00:10 PM
>>>>>>>>>>>>>>> Subject: Re: [conduit] How's your GFS?

>>>>>>>>>>>>>>> Hi Pete, et al.:

>>>>>>>>>>>>>>> We here at UAlbany continue to get spotty reception of the GFS 
>>>>>>>>>>>>>>> since 00Z today â
>>>>>>>>>>>>>>> anyone else having issues? We feed from Madison and State 
>>>>>>>>>>>>>>> College.

>>>>>>>>>>>>>>> Earlier thread below:

>>>>>>>>>>>>>>> -------------------------

>>>>>>>>>>>>>>> Yeah, Iâm not surprised that the addition of the  deg GFS is 
>>>>>>>>>>>>>>> causing the need
>>>>>>>>>>>>>>> for a bigger queue (and likely burlier machine). Thatâs the 
>>>>>>>>>>>>>>> main reason I have
>>>>>>>>>>>>>>> resisted requesting it.

>>>>>>>>>>>>>>> Iâll fix the issue that makes ldmstats show âinstall.atmosââ 
>>>>>>>>>>>>>>> instead of
>>>>>>>>>>>>>>> âcascade.atmosââ

>>>>>>>>>>>>>>> Something else must be at play since the  GFS has been flowing 
>>>>>>>>>>>>>>> for several
>>>>>>>>>>>>>>> weeks now without incident, likely tied to the increased 
>>>>>>>>>>>>>>> latency you starting
>>>>>>>>>>>>>>> seeing.

>>>>>>>>>>>>>>> Looks like we only got the GFS through 60 hours today with the 
>>>>>>>>>>>>>>> 12Z run, so
>>>>>>>>>>>>>>> something definitely appears to be amiss â Iâll cc: the conduit 
>>>>>>>>>>>>>>> list to see if
>>>>>>>>>>>>>>> anyone else is noticing problems.

>>>>>>>>>>>>>>> _____________________________________________
>>>>>>>>>>>>>>> Kevin Tyle, Systems Administrator
>>>>>>>>>>>>>>> Dept. of Atmospheric & Environmental Sciences
>>>>>>>>>>>>>>> University at Albany
>>>>>>>>>>>>>>> Earth Science 235, 1400 Washington Avenue
>>>>>>>>>>>>>>> Albany, NY 12222
>>>>>>>>>>>>>>> Email: ktyle@xxxxxxxxxx
>>>>>>>>>>>>>>> Phone: 518-442-4578
>>>>>>>>>>>>>>> _____________________________________________

>>>>>>>>>>>>>>> From: Pete Pokrandt [mailto: poker@xxxxxxxxxxxx ]
>>>>>>>>>>>>>>> Sent: Tuesday, September 08, 2015 12:17 PM
>>>>>>>>>>>>>>> To: Tyle, Kevin R < ktyle@xxxxxxxxxx >
>>>>>>>>>>>>>>> Cc: Daes Support < daessupport@xxxxxxxxxx >
>>>>>>>>>>>>>>> Subject: Re: How's your GFS?

>>>>>>>>>>>>>>> My GFS appears to be complete, but I do see that something's 
>>>>>>>>>>>>>>> going on with our
>>>>>>>>>>>>>>> feed - the latencies jumped way up somewhere over the weekend:

>>>>>>>>>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu

>>>>>>>>>>>>>>> You're seeing the same, and increased yet from your feed from 
>>>>>>>>>>>>>>> Penn State (at
>>>>>>>>>>>>>>> least to the machine ' install.atmos.albany.edu - I don't see 
>>>>>>>>>>>>>>> any stats
>>>>>>>>>>>>>>> reported from cascade, which is what it looks like you are 
>>>>>>>>>>>>>>> feeding from me on)

>>>>>>>>>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+install.atmos.albany.edu

>>>>>>>>>>>>>>> I think I need to buy more memory and keep a larger queue on 
>>>>>>>>>>>>>>> idd.aos.wisc.edu
>>>>>>>>>>>>>>> with the 0.25 deg GFS coming in. There are times where my queue 
>>>>>>>>>>>>>>> only holds
>>>>>>>>>>>>>>> about 20 minutes of data, which is likely contributing to your 
>>>>>>>>>>>>>>> incomplete GFS
>>>>>>>>>>>>>>> files..

>>>>>>>>>>>>>>> Here's what my 0.5 deg (the gblav2.* files) and the 0.25 deg 
>>>>>>>>>>>>>>> (gblav0p25 out to
>>>>>>>>>>>>>>> 87 h) look like for the 00 ant 06 UTC runs today

>>>>>>>>>>>>>>> 0.5 deg:
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 60953687 Sep 7 22:25 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F000
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 66996066 Sep 7 22:28 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F003
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 67902041 Sep 7 22:30 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F006
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 67961293 Sep 7 22:32 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F009
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68081826 Sep 7 22:35 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F012
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68710398 Sep 7 22:35 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F015
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69664268 Sep 7 22:36 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F018
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69177180 Sep 7 22:38 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F021
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69816235 Sep 7 22:38 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F024
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69010253 Sep 7 22:39 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F027
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69786985 Sep 7 22:40 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F030
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68876266 Sep 7 22:41 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F033
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69376601 Sep 7 22:42 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F036
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69029846 Sep 7 22:43 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F039
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69142392 Sep 7 22:44 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F042
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68990399 Sep 7 22:45 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F045
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69343366 Sep 7 22:46 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F048
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69150894 Sep 7 22:47 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F051
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69504675 Sep 7 22:47 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F054
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69196832 Sep 7 22:48 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F057
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69335487 Sep 7 22:50 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F060
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69261676 Sep 7 22:50 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F063
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69166068 Sep 7 22:51 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F066
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69054105 Sep 7 22:53 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F069
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68895264 Sep 7 22:54 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F072
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69202038 Sep 7 22:56 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F075
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69339334 Sep 7 22:56 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F078
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69181930 Sep 7 22:57 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F081
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69674148 Sep 7 22:58 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F084
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69383769 Sep 7 22:58 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F087
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69645526 Sep 7 22:59 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F090
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69119323 Sep 7 23:00 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F093
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69363296 Sep 7 23:01 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F096
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69030287 Sep 7 23:03 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F099
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69819322 Sep 7 23:03 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F102
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69498561 Sep 7 23:04 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F105
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69690447 Sep 7 23:05 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F108
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69274213 Sep 7 23:06 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F111
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70089206 Sep 7 23:07 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F114
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70007688 Sep 7 23:08 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F117
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70237308 Sep 7 23:08 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F120
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69849708 Sep 7 23:09 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F123
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69883550 Sep 7 23:11 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F126
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69586365 Sep 7 23:11 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F129
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70110782 Sep 7 23:12 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F132
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69430545 Sep 7 23:13 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F135
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69461630 Sep 7 23:14 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F138
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69264487 Sep 7 23:15 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F141
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69553206 Sep 7 23:16 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F144
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68924371 Sep 7 23:17 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F147
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69191965 Sep 7 23:17 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F150
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68639462 Sep 7 23:19 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F153
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69035706 Sep 7 23:22 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F156
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68831618 Sep 7 23:25 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F159
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69428952 Sep 7 23:27 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F162
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69514672 Sep 7 23:28 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F165
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69614097 Sep 7 23:29 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F168
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69404524 Sep 7 23:29 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F171
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69534566 Sep 7 23:30 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F174
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69528455 Sep 7 23:31 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F177
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69747643 Sep 7 23:31 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F180
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69397125 Sep 7 23:32 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F183
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69973323 Sep 7 23:32 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F186
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69070113 Sep 7 23:33 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F189
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69586837 Sep 7 23:34 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F192
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69202267 Sep 7 23:34 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F195
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69169373 Sep 7 23:35 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F198
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68193948 Sep 7 23:36 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F201
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 67963148 Sep 7 23:38 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F204
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 67689203 Sep 7 23:39 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F207
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68079977 Sep 7 23:41 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F210
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68931672 Sep 7 23:43 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F213
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68749459 Sep 7 23:46 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F216
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68739072 Sep 7 23:46 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F219
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68789427 Sep 7 23:47 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F222
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68031035 Sep 7 23:48 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F225
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68735199 Sep 7 23:48 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F228
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 65347330 Sep 7 23:49 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F231
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 65891902 Sep 7 23:49 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F234
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 65383729 Sep 7 23:50 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F237
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 66299227 Sep 7 23:50 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F240
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64525715 Sep 7 23:52 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F252
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64515690 Sep 7 23:53 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F264
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 63803271 Sep 7 23:53 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F276
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 63261621 Sep 7 23:54 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F288
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64171542 Sep 7 23:54 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F300
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64308576 Sep 7 23:56 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F312
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64334459 Sep 7 23:58 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F324
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64189700 Sep 7 23:59 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F336
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 63829248 Sep 7 23:59 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F348
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64655803 Sep 8 00:00 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F360
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64436657 Sep 8 00:07 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F372
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64546095 Sep 8 00:12 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090800_F384
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 61169101 Sep 8 04:26 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F000
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 67422108 Sep 8 04:28 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F003
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68374534 Sep 8 04:31 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F006
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68543418 Sep 8 04:33 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F009
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69298218 Sep 8 04:35 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F012
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69188133 Sep 8 04:36 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F015
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69917655 Sep 8 04:37 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F018
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69558566 Sep 8 04:38 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F021
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69659459 Sep 8 04:38 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F024
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69288102 Sep 8 04:40 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F027
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68686968 Sep 8 04:40 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F030
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68640234 Sep 8 04:42 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F033
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69544506 Sep 8 04:42 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F036
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68462036 Sep 8 04:43 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F039
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69287354 Sep 8 04:44 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F042
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69228412 Sep 8 04:45 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F045
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69444769 Sep 8 04:46 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F048
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69089036 Sep 8 04:47 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F051
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69542812 Sep 8 04:48 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F054
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69377775 Sep 8 04:49 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F057
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69324867 Sep 8 04:50 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F060
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69313464 Sep 8 04:51 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F063
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69820155 Sep 8 04:52 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F066
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69484687 Sep 8 04:52 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F069
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69581997 Sep 8 04:53 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F072
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69189693 Sep 8 04:54 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F075
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69751906 Sep 8 04:55 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F078
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69558875 Sep 8 04:56 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F081
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69903084 Sep 8 04:58 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F084
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69627748 Sep 8 04:59 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F087
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69678696 Sep 8 04:59 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F090
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69497446 Sep 8 05:00 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F093
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69735442 Sep 8 05:01 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F096
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69767861 Sep 8 05:02 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F099
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70169785 Sep 8 05:03 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F102
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69625644 Sep 8 05:04 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F105
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69954293 Sep 8 05:05 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F108
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69996186 Sep 8 05:06 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F111
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70297897 Sep 8 05:06 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F114
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70037957 Sep 8 05:08 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F117
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69968183 Sep 8 05:08 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F120
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69564905 Sep 8 05:10 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F123
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69725865 Sep 8 05:11 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F126
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69349475 Sep 8 05:11 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F129
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69625604 Sep 8 05:12 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F132
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69392152 Sep 8 05:15 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F135
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69551134 Sep 8 05:18 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F138
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69108820 Sep 8 05:19 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F141
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69469618 Sep 8 05:19 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F144
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 68774645 Sep 8 05:20 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F147
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69135260 Sep 8 05:20 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F150
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69009857 Sep 8 05:21 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F153
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69647753 Sep 8 05:21 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F156
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69604259 Sep 8 05:22 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F159
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69851358 Sep 8 05:22 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F162
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69621423 Sep 8 05:23 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F165
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69987289 Sep 8 05:24 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F168
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70009168 Sep 8 05:24 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F171
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70272431 Sep 8 05:25 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F174
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69951044 Sep 8 05:26 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F177
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70294466 Sep 8 05:28 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F180
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69693077 Sep 8 05:31 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F183
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70277595 Sep 8 05:35 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F186
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70161497 Sep 8 05:36 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F189
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70075264 Sep 8 05:37 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F192
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69929971 Sep 8 05:37 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F195
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69879151 Sep 8 05:38 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F198
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69726455 Sep 8 05:38 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F201
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70186834 Sep 8 05:38 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F204
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69735649 Sep 8 05:39 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F207
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70062469 Sep 8 05:40 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F210
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69475211 Sep 8 05:40 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F213
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69688060 Sep 8 05:41 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F216
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69169089 Sep 8 05:42 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F219
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69623322 Sep 8 05:42 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F222
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69434126 Sep 8 05:43 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F225
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69447710 Sep 8 05:44 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F228
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69232930 Sep 8 05:44 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F231
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69688395 Sep 8 05:45 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F234
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 69476983 Sep 8 05:47 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F237
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 70027781 Sep 8 05:50 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F240
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64748968 Sep 8 05:52 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F252
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64729059 Sep 8 05:53 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F264
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64211460 Sep 8 05:53 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F276
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64117374 Sep 8 05:54 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F288
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64123032 Sep 8 05:54 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F300
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64714736 Sep 8 05:56 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F312
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 65052210 Sep 8 05:56 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F324
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 65123631 Sep 8 05:57 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F336
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64903451 Sep 8 05:58 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F348
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64423290 Sep 8 05:59 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F360
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 64365594 Sep 8 05:59 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F372
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 63855749 Sep 8 06:07 
>>>>>>>>>>>>>>> /data/grib2/gblav2.15090806_F384

>>>>>>>>>>>>>>> 0.25 deg

>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 180752345 Sep 7 22:25 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_Fanl
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 197882387 Sep 7 22:26 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F000
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 217304897 Sep 7 22:27 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F003
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 221447144 Sep 7 22:30 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F006
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 221383770 Sep 7 22:32 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F009
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 222748480 Sep 7 22:34 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F012
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 224209489 Sep 7 22:35 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F015
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226360332 Sep 7 22:36 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F018
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225185199 Sep 7 22:37 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F021
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226720828 Sep 7 22:38 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F024
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 224211990 Sep 7 22:38 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F027
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226623368 Sep 7 22:40 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F030
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 224601041 Sep 7 22:41 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F033
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225696377 Sep 7 22:41 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F036
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 224803488 Sep 7 22:43 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F039
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225463303 Sep 7 22:44 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F042
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 224172234 Sep 7 22:44 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F045
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225750651 Sep 7 22:45 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F048
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 224513834 Sep 7 22:46 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F051
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225871134 Sep 7 22:47 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F054
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 224871484 Sep 7 22:48 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F057
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225954437 Sep 7 22:50 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F060
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225600052 Sep 7 22:50 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F063
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225672348 Sep 7 22:51 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F066
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225064451 Sep 7 22:52 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F069
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225318101 Sep 7 22:54 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F072
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225303961 Sep 7 22:57 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F075
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226805528 Sep 7 23:03 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F078
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226187062 Sep 7 23:08 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F081
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 227313364 Sep 7 23:09 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F084
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226221831 Sep 7 23:15 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090800_F087
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 197951753 Sep 8 04:25 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F000
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 181438882 Sep 8 04:26 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_Fanl
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 218273142 Sep 8 04:28 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F003
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 222180270 Sep 8 04:30 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F006
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 222627637 Sep 8 04:32 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F009
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225440960 Sep 8 04:34 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F012
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 224877734 Sep 8 04:35 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F015
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226700650 Sep 8 04:36 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F018
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225325799 Sep 8 04:37 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F021
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226163438 Sep 8 04:38 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F024
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225234793 Sep 8 04:39 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F027
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 224315172 Sep 8 04:40 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F030
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 223485303 Sep 8 04:41 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F033
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226101395 Sep 8 04:42 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F036
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 222880336 Sep 8 04:43 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F039
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225276943 Sep 8 04:44 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F042
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225167793 Sep 8 04:45 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F045
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225771493 Sep 8 04:46 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F048
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225066649 Sep 8 04:47 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F051
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225905191 Sep 8 04:47 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F054
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225706912 Sep 8 04:48 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F057
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225891555 Sep 8 04:49 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F060
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225723607 Sep 8 04:50 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F063
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 227329359 Sep 8 04:51 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F066
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226381130 Sep 8 04:52 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F069
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 227000926 Sep 8 04:53 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F072
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 225483067 Sep 8 04:54 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F075
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 227295269 Sep 8 04:55 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F078
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226316715 Sep 8 04:55 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F081
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 227632093 Sep 8 04:57 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F084
>>>>>>>>>>>>>>> -rw-r--r--. 1 ldm ldm 226447758 Sep 8 05:01 
>>>>>>>>>>>>>>> /data/grib2/gblav0p25.15090806_F087

>>>>>>>>>>>>>>> Pete

>>>>>>>>>>>>>>> On 09/08/2015 08:43 AM, Tyle, Kevin R wrote:

>>>>>>>>>>>>>>>> Hi Pete,

>>>>>>>>>>>>>>>> Weâve had incomplete GFS the last two runs (00 and 06 UTC 
>>>>>>>>>>>>>>>> today) â how did
>>>>>>>>>>>>>>>> things look on your end?

>>>>>>>>>>>>>>>> Thanks,

>>>>>>>>>>>>>>>> Kevin

>>>>>>>>>>>>>>>> _____________________________________________
>>>>>>>>>>>>>>>> Kevin Tyle, Systems Administrator
>>>>>>>>>>>>>>>> Dept. of Atmospheric & Environmental Sciences
>>>>>>>>>>>>>>>> University at Albany
>>>>>>>>>>>>>>>> Earth Science 235, 1400 Washington Avenue
>>>>>>>>>>>>>>>> Albany, NY 12222
>>>>>>>>>>>>>>>> Email: ktyle@xxxxxxxxxx
>>>>>>>>>>>>>>>> Phone: 518-442-4578
>>>>>>>>>>>>>>>> _____________________________________________

>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Pete Pokrandt - Systems Programmer
>>>>>>>>>>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>>>>>>>>>>>> 608-262-3086 - poker@xxxxxxxxxxxx

>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> conduit mailing list
>>>>>>>>>>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>>>>>>>>>>> For list information or to unsubscribe, visit:
>>>>>>>>>>>>>>> http://www.unidata.ucar.edu/mailing_lists/

>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Arthur A. Person
>>>>>>>>>>>>>> Research Assistant, System Administrator
>>>>>>>>>>>>>> Penn State Department of Meteorology
>>>>>>>>>>>>>> email: aap1@xxxxxxx , phone: 814-863-1563

>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> conduit mailing list
>>>>>>>>>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>>>>>>>>>> For list information or to unsubscribe, visit:
>>>>>>>>>>>>>> http://www.unidata.ucar.edu/mailing_lists/

>>>>>>>>>>>> --
>>>>>>>>>>>> Arthur A. Person
>>>>>>>>>>>> Research Assistant, System Administrator
>>>>>>>>>>>> Penn State Department of Meteorology
>>>>>>>>>>>> email: aap1@xxxxxxx , phone: 814-863-1563

>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> conduit mailing list
>>>>>>>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>>>>>>>> For list information or to unsubscribe, visit:
>>>>>>>>>>>> http://www.unidata.ucar.edu/mailing_lists/

>>>>>>>>>>> --
>>>>>>>>>>> Arthur A. Person
>>>>>>>>>>> Research Assistant, System Administrator
>>>>>>>>>>> Penn State Department of Meteorology
>>>>>>>>>>> email: aap1@xxxxxxx , phone: 814-863-1563

>>>>>>>>>> --
>>>>>>>>>> Arthur A. Person
>>>>>>>>>> Research Assistant, System Administrator
>>>>>>>>>> Penn State Department of Meteorology
>>>>>>>>>> email: aap1@xxxxxxx , phone: 814-863-1563

>>>>>>>>> --
>>>>>>>>> Arthur A. Person
>>>>>>>>> Research Assistant, System Administrator
>>>>>>>>> Penn State Department of Meteorology
>>>>>>>>> email: aap1@xxxxxxx , phone: 814-863-1563

>>>>>>>> _______________________________________________
>>>>>>>> conduit mailing list
>>>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>>>> For list information or to unsubscribe, visit:
>>>>>>>> http://www.unidata.ucar.edu/mailing_lists/

>>>>>> _______________________________________________
>>>>>> conduit mailing list
>>>>>> conduit@xxxxxxxxxxxxxxxx
>>>>>> For list information or to unsubscribe, visit:
>>>>>> http://www.unidata.ucar.edu/mailing_lists/

>>>>> --
>>>>> Arthur A. Person
>>>>> Research Assistant, System Administrator
>>>>> Penn State Department of Meteorology
>>>>> email: aap1@xxxxxxx , phone: 814-863-1563

>>> --
>>> Arthur A. Person
>>> Research Assistant, System Administrator
>>> Penn State Department of Meteorology
>>> email: aap1@xxxxxxx , phone: 814-863-1563

>>> _______________________________________________
>>> conduit mailing list
>>> conduit@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/

>> _______________________________________________
>> Ncep.list.pmb-dataflow mailing list 
>> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow

-- 
Arthur A. Person 
Research Assistant, System Administrator 
Penn State Department of Meteorology 
email: aap1@xxxxxxx, phone: 814-863-1563