Re: [ldm-users] NEXRAD2/NEXRAD3 lag?

And it's ext4.

This is a new problem...it started about three weeks ago.  For the two
years prior to that, we saw NO delays *ever*.  And, as I mentioned, others
like iastate also have observed this, as have our upstream providers (who
are getting their data from the tier 1 servers).


On Sat, May 17, 2014 at 4:06 AM, Blair Trosper <
blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:

> And it's Debian 7
>
>
> On Sat, May 17, 2014 at 4:06 AM, Blair Trosper <
> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>
>> Yes.  SAS drives on all three machines, two are Intel Xeon Ivy
>> Bridge...the other is an AMD hexacore running at 3.6 GHz.  Each box has
>> 1Gbps symmetric and at least 16GB RAM.
>>
>>
>> On Sat, May 17, 2014 at 3:33 AM, Robert Mullenax <
>> Robert.Mullenax@xxxxxxxxxxxxx> wrote:
>>
>>> Are you 100% sure it isn't a local write to disk issue..not a big enough
>>> queue or general performance issues. What OS, file system, and type of
>>> disks do you have?
>>>
>>>
>>>
>>> On May 17, 2014, at 12:45 AM, "Blair Trosper" <
>>> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>>>
>>> Odd.  We watch the Internet2 weathermap in real time when we see delay,
>>> and none of the 10Gbps connections are ever more than 1% utilized.  So it's
>>> not a problem inside Internet2.
>>>
>>> Can someone help me remember where the document is that explains how
>>> data gets from NOAA into the LDM feed trees?
>>>
>>> The last one I saw was a few years ago, and I absolutely cannot remember
>>> how I found it.  Unfortunately, as with a lot of things on the UNIDATA site
>>> (like rtstats), there are often broken links or server errors -- or
>>> documents that are just flat our wrong or out of date as well.
>>>
>>>
>>> On Sat, May 17, 2014 at 1:38 AM, Ryan Hickman <ryan@xxxxxxxxxxxxxxxx>wrote:
>>>
>>>> We (AllisonHouse) do not acquire the NEXRAD2 data outside of LDM. In
>>>> fact, we share common upstream providers.
>>>>
>>>> This sounds like an issue that is unique to Internet2.
>>>>
>>>> - Ryan Hickman
>>>> Chief Technology Officer
>>>> AllisonHouse
>>>>
>>>> On May 17, 2014, at 12:18 AM, Blair Trosper <
>>>> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>>>>
>>>> No joy there.  Our providers (TAMU, OU, WISC, et al) are all under tier
>>>> 1 providers, and there is no lag.
>>>>
>>>> I can't find the document that describes how the NEXRAD2 data is fed
>>>> into the LDM system.  I think it happens in DC at the Internet2 aperture,
>>>> but that information may be out of date (or I may be remembering it wrong).
>>>>
>>>> As I recall, two weeks ago, we weren't the only ones seeing this lag.
>>>>
>>>> And we're still seeing it on three servers with three separate upstream
>>>> providers -- and each server is deliberately on different BGP/backbone
>>>> providers for geographic AND network diversity.
>>>>
>>>> Something's not quite right, and I think the key is understanding how
>>>> the NEXRAD2 data gets into LDM...and then going to THAT source to find out
>>>> why it's delayed.  I know Allison House's data isn't delayed (whereas
>>>> iastate's is), so my guess is they're acquiring it independently of LDM.
>>>>
>>>> So we know iastate.edu is getting data 10 minutes late just like we
>>>> are, as are our upstream providers...but the servers themselves are always
>>>> at 0 or 1 second of lag.
>>>>
>>>> Something is definitely wrong.  We don't see any lag on any of the
>>>> other feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3, etc).  This seems
>>>> to be unique to NEXRAD2.
>>>>
>>>> Can someone from UCAR/UNIDATA jump in and help us...or should we open a
>>>> formal support ticket?
>>>>
>>>>
>>>> On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson <
>>>> emmerson.steven@xxxxxxxxx> wrote:
>>>>
>>>>> It could be that the last NEXRAD2 data-product is stuck in a system
>>>>> buffer. Try putting a "-flush" or "-close" on the relevant PIPE action.
>>>>> On May 11, 2014 11:49 AM, "Gilbert Sebenste" <
>>>>> sebenste@xxxxxxxxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>>> Hi Blair,
>>>>>>
>>>>>>  We are still seeing this.  I know the TOC was made aware of this
>>>>>>> issue two
>>>>>>> weeks ago, but despite all our servers being 0 seconds lagged on the
>>>>>>> NEXRAD2 feed tree, our data (from multiple providers who are also not
>>>>>>> lagged) is at least 10 minutes behind.
>>>>>>>
>>>>>>
>>>>>> Been off the grid, for the most part, or I would have answered
>>>>>> earlier. Things to look for:
>>>>>>
>>>>>> Can you do a traceroute to the servers in question? See if anything is
>>>>>> jammed up inbetween them and you.
>>>>>>
>>>>>> Are all network cards set at least at 100 mb? If 10 mb, it can't
>>>>>> handle it.
>>>>>>
>>>>>> Is the system clock correct? Are you using ntp? This is a must.
>>>>>>
>>>>>>  Also, this page is a 500 internal server error and has been for
>>>>>>> months, and nobody has fixed it or seems to even notice:
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_
>>>>>>> topogif?NEXRAD2
>>>>>>>
>>>>>>> And this one takes over a minute to load:
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_
>>>>>>> feedtree?NEXRAD2
>>>>>>>
>>>>>>> ...but you can verify from the second link that we're not lagging
>>>>>>> (search
>>>>>>> for "tampa" or "chicago").
>>>>>>>
>>>>>>
>>>>>> Give those a try aboove, and see what happens.
>>>>>>
>>>>>> Gilbert
>>>>>>
>>>>>> ************************************************************
>>>>>> *******************
>>>>>> Gilbert Sebenste
>>>>>>  ********
>>>>>> (My opinions only!)
>>>>>>  ******
>>>>>> Staff Meteorologist, Northern Illinois University
>>>>>>  ****
>>>>>> E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx
>>>>>>    ***
>>>>>> web: http://weather.admin.niu.edu
>>>>>>    **
>>>>>> Twitter: http://www.twitter.com/NIU_Weather
>>>>>>    **
>>>>>> Facebook: http://www.facebook.com/niu.weather
>>>>>>     *
>>>>>> ************************************************************
>>>>>> *******************
>>>>>>
>>>>>> _______________________________________________
>>>>>> ldm-users mailing list
>>>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>>>> For list information or to unsubscribe,  visit:
>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ldm-users mailing list
>>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>>> For list information or to unsubscribe,  visit:
>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>>>>  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>>>>  ∙
>>>> Blair Trosper
>>>>   Updraft Networks / Weather Data
>>>>   NOC:  469-844-5440
>>>>   Early Watch Notifications:  http://twitter.com/weatherwatches
>>>>
>>>> _______________________________________________
>>>> ldm-users mailing list
>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>>>  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>>> Blair Trosper
>>>   Updraft Networks / Weather Data
>>>   NOC:  469-844-5440
>>>   Early Watch Notifications:  http://twitter.com/weatherwatches
>>>
>>> _______________________________________________
>>> ldm-users mailing list
>>> ldm-users@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe,  visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>>
>>
>>
>> --
>>
>> ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>>  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>> Blair Trosper
>>   Updraft Networks / Weather Data
>>   NOC:  469-844-5440
>>   Early Watch Notifications:  http://twitter.com/weatherwatches
>>
>
>
>
> --
>
> ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
> Blair Trosper
>   Updraft Networks / Weather Data
>   NOC:  469-844-5440
>   Early Watch Notifications:  http://twitter.com/weatherwatches
>



-- 

∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
 ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
Blair Trosper
  Updraft Networks / Weather Data
  NOC:  469-844-5440
  Early Watch Notifications:  http://twitter.com/weatherwatches
  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: