Daryl @ IAState.edu would be one of the other ones who's noticed this off
the top of my head. Maybe he could chime in from two or so weeks ago when
this first started.
I'm keen at this point to point out that our servers and Daryl's share NO
upstream peers in common whatsoever. Completely different and discrete on
the topology for NEXRAD2.
On Sun, May 18, 2014 at 8:24 PM, Blair Trosper <
blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
> We're not the only ones seeing it...and again, all our servers are 1Gbps
> using less than 1% of their bandwidth.
>
> It's also for ALL sites, not any specific WFO/SRH. I've checked with at
> least two of our current providers (who pull from the very top of the
> topology, and they confirm the issue).
>
> Karen: Can you nominate an upstream server for one of ours to use so we
> can test this out? Can we be fed from a tier 1 server for a day or two to
> definitely rule that out?
>
> I might be with you if it were one server, but we're talking about gigabit
> servers on separate backbone providers with separate upstream providers in
> four geographically disparate locations...all with the *exact* same lag
> at any given moment. The odds of that are astronomical, especially when
> you factor in that one tier-1-minutes-one providers see the same thing.
>
>
> On Sat, May 17, 2014 at 9:35 AM, Karen Cooper - NOAA Affiliate <
> karen.cooper@xxxxxxxx> wrote:
>
>> The level 2 data is inserted into Ldm queues at the radar RPGs as it is
>> generated. If it were a problem there everyone would be seeing it.
>>
>> Sent from me! Please pardon any tpyos
>>
>>
>> On May 17, 2014, at 1:18 AM, Blair Trosper <
>> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>>
>> No joy there. Our providers (TAMU, OU, WISC, et al) are all under tier 1
>> providers, and there is no lag.
>>
>> I can't find the document that describes how the NEXRAD2 data is fed into
>> the LDM system. I think it happens in DC at the Internet2 aperture, but
>> that information may be out of date (or I may be remembering it wrong).
>>
>> As I recall, two weeks ago, we weren't the only ones seeing this lag.
>>
>> And we're still seeing it on three servers with three separate upstream
>> providers -- and each server is deliberately on different BGP/backbone
>> providers for geographic AND network diversity.
>>
>> Something's not quite right, and I think the key is understanding how the
>> NEXRAD2 data gets into LDM...and then going to THAT source to find out why
>> it's delayed. I know Allison House's data isn't delayed (whereas iastate's
>> is), so my guess is they're acquiring it independently of LDM.
>>
>> So we know iastate.edu is getting data 10 minutes late just like we are,
>> as are our upstream providers...but the servers themselves are always at 0
>> or 1 second of lag.
>>
>> Something is definitely wrong. We don't see any lag on any of the other
>> feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3, etc). This seems to be
>> unique to NEXRAD2.
>>
>> Can someone from UCAR/UNIDATA jump in and help us...or should we open a
>> formal support ticket?
>>
>>
>> On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson <
>> emmerson.steven@xxxxxxxxx> wrote:
>>
>>> It could be that the last NEXRAD2 data-product is stuck in a system
>>> buffer. Try putting a "-flush" or "-close" on the relevant PIPE action.
>>> On May 11, 2014 11:49 AM, "Gilbert Sebenste" <
>>> sebenste@xxxxxxxxxxxxxxxxxxxxx> wrote:
>>>
>>>> Hi Blair,
>>>>
>>>> We are still seeing this. I know the TOC was made aware of this issue
>>>>> two
>>>>> weeks ago, but despite all our servers being 0 seconds lagged on the
>>>>> NEXRAD2 feed tree, our data (from multiple providers who are also not
>>>>> lagged) is at least 10 minutes behind.
>>>>>
>>>>
>>>> Been off the grid, for the most part, or I would have answered earlier .
>>>> Things to look for:
>>>>
>>>> Can you do a traceroute to the servers in question? See if anything is
>>>> jammed up inbetween them and you.
>>>>
>>>> Are all network cards set at least at 100 mb? If 10 mb, it can't handle
>>>> it.
>>>>
>>>> Is the system clock correct? Are you using ntp? This is a must.
>>>>
>>>> Also, this page is a 500 internal server error and has been for
>>>>> months, and nobody has fixed it or seems to even notice:
>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_
>>>>> topogif?NEXRAD2
>>>>>
>>>>> And this one takes over a minute to load:
>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_
>>>>> feedtree?NEXRAD2
>>>>>
>>>>> ...but you can verify from the second link that we're not lagging
>>>>> (search
>>>>> for "tampa" or "chicago").
>>>>>
>>>>
>>>> Give those a try aboove, and see what happens.
>>>>
>>>> Gilbert
>>>>
>>>> ************************************************************
>>>> *******************
>>>> Gilbert Sebenste
>>>> ********
>>>> (My opinions only!)
>>>> ******
>>>> Staff Meteorologist, Northern Illinois University
>>>> ****
>>>> E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx
>>>> ***
>>>> web: http://weather.admin.niu.edu
>>>> **
>>>> Twitter: http://www.twitter.com/NIU_Weather
>>>> **
>>>> Facebook: http://www.facebook.com/niu.weather
>>>> *
>>>> ************************************************************
>>>> *******************
>>>>
>>>> _______________________________________________
>>>> ldm-users mailing list
>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
>>> _______________________________________________
>>> ldm-users mailing list
>>> ldm-users@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>
>>
>>
>> --
>>
>> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
>> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
>> Blair Trosper
>> Updraft Networks / Weather Data
>> NOC: 469-844-5440
>> Early Watch Notifications: http://twitter.com/weatherwatches
>>
>> _______________________________________________
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>>
>
>
> --
>
> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
> Blair Trosper
> Updraft Networks / Weather Data
> NOC: 469-844-5440
> Early Watch Notifications: http://twitter.com/weatherwatches
>
--
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
Blair Trosper
Updraft Networks / Weather Data
NOC: 469-844-5440
Early Watch Notifications: http://twitter.com/weatherwatches
