Re: [ldm-users] NEXRAD2/NEXRAD3 lag?

And it's Debian 7


On Sat, May 17, 2014 at 4:06 AM, Blair Trosper <
blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:

> Yes.  SAS drives on all three machines, two are Intel Xeon Ivy
> Bridge...the other is an AMD hexacore running at 3.6 GHz.  Each box has
> 1Gbps symmetric and at least 16GB RAM.
>
>
> On Sat, May 17, 2014 at 3:33 AM, Robert Mullenax <
> Robert.Mullenax@xxxxxxxxxxxxx> wrote:
>
>> Are you 100% sure it isn't a local write to disk issue..not a big enough
>> queue or general performance issues. What OS, file system, and type of
>> disks do you have?
>>
>>
>>
>> On May 17, 2014, at 12:45 AM, "Blair Trosper" <
>> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>>
>> Odd.  We watch the Internet2 weathermap in real time when we see delay,
>> and none of the 10Gbps connections are ever more than 1% utilized.  So it's
>> not a problem inside Internet2.
>>
>> Can someone help me remember where the document is that explains how data
>> gets from NOAA into the LDM feed trees?
>>
>> The last one I saw was a few years ago, and I absolutely cannot remember
>> how I found it.  Unfortunately, as with a lot of things on the UNIDATA site
>> (like rtstats), there are often broken links or server errors -- or
>> documents that are just flat our wrong or out of date as well.
>>
>>
>> On Sat, May 17, 2014 at 1:38 AM, Ryan Hickman <ryan@xxxxxxxxxxxxxxxx>wrote:
>>
>>> We (AllisonHouse) do not acquire the NEXRAD2 data outside of LDM. In
>>> fact, we share common upstream providers.
>>>
>>> This sounds like an issue that is unique to Internet2.
>>>
>>> - Ryan Hickman
>>> Chief Technology Officer
>>> AllisonHouse
>>>
>>> On May 17, 2014, at 12:18 AM, Blair Trosper <
>>> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>>>
>>> No joy there.  Our providers (TAMU, OU, WISC, et al) are all under tier
>>> 1 providers, and there is no lag.
>>>
>>> I can't find the document that describes how the NEXRAD2 data is fed
>>> into the LDM system.  I think it happens in DC at the Internet2 aperture,
>>> but that information may be out of date (or I may be remembering it wrong).
>>>
>>> As I recall, two weeks ago, we weren't the only ones seeing this lag.
>>>
>>> And we're still seeing it on three servers with three separate upstream
>>> providers -- and each server is deliberately on different BGP/backbone
>>> providers for geographic AND network diversity.
>>>
>>> Something's not quite right, and I think the key is understanding how
>>> the NEXRAD2 data gets into LDM...and then going to THAT source to find out
>>> why it's delayed.  I know Allison House's data isn't delayed (whereas
>>> iastate's is), so my guess is they're acquiring it independently of LDM .
>>>
>>> So we know iastate.edu is getting data 10 minutes late just like we
>>> are, as are our upstream providers...but the servers themselves are always
>>> at 0 or 1 second of lag.
>>>
>>> Something is definitely wrong.  We don't see any lag on any of the other
>>> feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3, etc).  This seems to be
>>> unique to NEXRAD2.
>>>
>>> Can someone from UCAR/UNIDATA jump in and help us...or should we open a
>>> formal support ticket?
>>>
>>>
>>> On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson <
>>> emmerson.steven@xxxxxxxxx> wrote:
>>>
>>>> It could be that the last NEXRAD2 data-product is stuck in a system
>>>> buffer. Try putting a "-flush" or "-close" on the relevant PIPE action .
>>>> On May 11, 2014 11:49 AM, "Gilbert Sebenste" <
>>>> sebenste@xxxxxxxxxxxxxxxxxxxxx> wrote:
>>>>
>>>>> Hi Blair,
>>>>>
>>>>>  We are still seeing this.  I know the TOC was made aware of this
>>>>>> issue two
>>>>>> weeks ago, but despite all our servers being 0 seconds lagged on the
>>>>>> NEXRAD2 feed tree, our data (from multiple providers who are also not
>>>>>> lagged) is at least 10 minutes behind.
>>>>>>
>>>>>
>>>>> Been off the grid, for the most part, or I would have answered
>>>>> earlier. Things to look for:
>>>>>
>>>>> Can you do a traceroute to the servers in question? See if anything is
>>>>> jammed up inbetween them and you.
>>>>>
>>>>> Are all network cards set at least at 100 mb? If 10 mb, it can't
>>>>> handle it.
>>>>>
>>>>> Is the system clock correct? Are you using ntp? This is a must.
>>>>>
>>>>>  Also, this page is a 500 internal server error and has been for
>>>>>> months, and nobody has fixed it or seems to even notice:
>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_
>>>>>> topogif?NEXRAD2
>>>>>>
>>>>>> And this one takes over a minute to load:
>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_
>>>>>> feedtree?NEXRAD2
>>>>>>
>>>>>> ...but you can verify from the second link that we're not lagging
>>>>>> (search
>>>>>> for "tampa" or "chicago").
>>>>>>
>>>>>
>>>>> Give those a try aboove, and see what happens.
>>>>>
>>>>> Gilbert
>>>>>
>>>>> ************************************************************
>>>>> *******************
>>>>> Gilbert Sebenste
>>>>>  ********
>>>>> (My opinions only!)
>>>>>  ******
>>>>> Staff Meteorologist, Northern Illinois University
>>>>>  ****
>>>>> E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx
>>>>>    ***
>>>>> web: http://weather.admin.niu.edu
>>>>>  **
>>>>> Twitter: http://www.twitter.com/NIU_Weather
>>>>>  **
>>>>> Facebook: http://www.facebook.com/niu.weather
>>>>>   *
>>>>> ************************************************************
>>>>> *******************
>>>>>
>>>>> _______________________________________________
>>>>> ldm-users mailing list
>>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>>> For list information or to unsubscribe,  visit:
>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>
>>>>
>>>> _______________________________________________
>>>> ldm-users mailing list
>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>>>  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>>> Blair Trosper
>>>   Updraft Networks / Weather Data
>>>   NOC:  469-844-5440
>>>   Early Watch Notifications:  http://twitter.com/weatherwatches
>>>
>>> _______________________________________________
>>> ldm-users mailing list
>>> ldm-users@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe,  visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>>
>>
>>
>> --
>>
>> ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>>  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>> Blair Trosper
>>   Updraft Networks / Weather Data
>>   NOC:  469-844-5440
>>   Early Watch Notifications:  http://twitter.com/weatherwatches
>>
>> _______________________________________________
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>>
>
>
> --
>
> ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
>  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
> Blair Trosper
>   Updraft Networks / Weather Data
>   NOC:  469-844-5440
>   Early Watch Notifications:  http://twitter.com/weatherwatches
>



-- 

∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
 ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙  ∙
Blair Trosper
  Updraft Networks / Weather Data
  NOC:  469-844-5440
  Early Watch Notifications:  http://twitter.com/weatherwatches
  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: