Re: [ldm-users] NEXRAD2/NEXRAD3 lag?

Blair,

Get me your IP address and you you can try feeding NEXRAD2 from idd.aos.wisc.edu for a while. I've got virtually no lag right now.

[ldm@idd ~]$ ldmadmin watch -f nexrad2
(Type ^D when finished)
May 19 01:40:33 pqutil INFO: 42811 20140519014031.935 NEXRAD2 464006 L2-BZIP2/KEMX/20140519013931/464/6/I/V06/0 May 19 01:40:33 pqutil INFO: 58507 20140519014031.601 NEXRAD2 41024 L2-BZIP2/KBBX/20140519013827/41/24/I/V07/0 May 19 01:40:33 pqutil INFO: 27886 20140519014031.993 NEXRAD2 23012 L2-BZIP2/KENX/20140519013811/23/12/I/V06/0 May 19 01:40:33 pqutil INFO: 18921 20140519014031.954 NEXRAD2 494036 L2-BZIP2/KBLX/20140519013620/494/36/I/V06/0 May 19 01:40:33 pqutil INFO: 70526 20140519014032.136 NEXRAD2 830026 L2-BZIP2/KDAX/20140519013922/830/26/I/V06/0 May 19 01:40:33 pqutil INFO: 26415 20140519014032.555 NEXRAD2 92015 L2-BZIP2/KLRX/20140519013923/92/15/I/V06/0 May 19 01:40:33 pqutil INFO: 75834 20140519014032.121 NEXRAD2 458021 L2-BZIP2/KMSX/20140519013927/458/21/I/V06/0 May 19 01:40:34 pqutil INFO: 40577 20140519014032.105 NEXRAD2 543070 L2-BZIP2/KTFX/20140519013553/543/70/E/V06/0 May 19 01:40:34 pqutil INFO: 10731 20140519014033.148 NEXRAD2 414001 L2-BZIP2/KUDX/20140519014030/414/1/S/V06/0 May 19 01:40:34 pqutil INFO: 51404 20140519014032.813 NEXRAD2 669025 L2-BZIP2/KMXX/20140519013530/669/25/I/V06/0 May 19 01:40:34 pqutil INFO: 43796 20140519014033.203 NEXRAD2 763047 L2-BZIP2/KGLD/20140519013743/763/47/I/V06/0 May 19 01:40:34 pqutil INFO: 7796 20140519014033.375 NEXRAD2 68001 L2-BZIP2/KMLB/20140519014030/68/1/S/V06/0 May 19 01:40:34 pqutil INFO: 40044 20140519014032.949 NEXRAD2 175025 L2-BZIP2/KSHV/20140519013530/175/25/I/V06/0 May 19 01:40:34 pqutil INFO: 60712 20140519014033.290 NEXRAD2 348018 L2-BZIP2/KDVN/20140519013657/348/18/I/V06/0 May 19 01:40:34 pqutil INFO: 83700 20140519014032.995 NEXRAD2 653030 L2-BZIP2/KFSD/20140519013912/653/30/I/V06/0 May 19 01:40:34 pqutil INFO: 63381 20140519014033.147 NEXRAD2 112031 L2-BZIP2/KCAE/20140519013831/112/31/I/V06/0 May 19 01:40:34 pqutil INFO: 8601 20140519014033.353 NEXRAD2 884011 L2-BZIP2/PHMO/20140519013841/884/11/I/V07/0 May 19 01:40:34 pqutil INFO: 40840 20140519014032.859 NEXRAD2 195017 L2-BZIP2/PHKM/20140519013747/195/17/I/V07/0 May 19 01:40:34 pqutil INFO: 71889 20140519014033.455 NEXRAD2 598014 L2-BZIP2/KEAX/20140519013747/598/14/I/V06/0 May 19 01:40:34 pqutil INFO: 90198 20140519014033.699 NEXRAD2 227013 L2-BZIP2/KMPX/20140519013802/227/13/I/V06/0 May 19 01:40:34 pqutil INFO: 27715 20140519014033.619 NEXRAD2 963033 L2-BZIP2/KDFX/20140519013728/963/33/I/V07/0 May 19 01:40:34 pqutil INFO: 120505 20140519014033.009 NEXRAD2 70027 L2-BZIP2/KCBW/20140519013800/70/27/I/V06/0 May 19 01:40:34 pqutil INFO: 6778 20140519014034.053 NEXRAD2 946001 L2-BZIP2/KAPX/20140519014031/946/1/S/V06/0 May 19 01:40:35 pqutil INFO: 64914 20140519014032.807 NEXRAD2 430026 L2-BZIP2/PACG/20140519013545/430/26/I/V06/0

Pete

On 5/18/2014 8:24 PM, Blair Trosper wrote:
We're not the only ones seeing it...and again, all our servers are 1Gbps using less than 1% of their bandwidth.

It's also for ALL sites, not any specific WFO/SRH. I've checked with at least two of our current providers (who pull from the very top of the topology, and they confirm the issue).

Karen: Can you nominate an upstream server for one of ours to use so we can test this out? Can we be fed from a tier 1 server for a day or two to definitely rule that out?

I might be with you if it were one server, but we're talking about gigabit servers on separate backbone providers with separate upstream providers in four geographically disparate locations...all with the */exact/* same lag at any given moment. The odds of that are astronomical, especially when you factor in that one tier-1-minutes-one providers see the same thing.


On Sat, May 17, 2014 at 9:35 AM, Karen Cooper - NOAA Affiliate <karen.cooper@xxxxxxxx <mailto:karen.cooper@xxxxxxxx>> wrote:

    The level 2 data is inserted into Ldm queues at the radar RPGs as
    it is generated.   If it were a problem there everyone would be
    seeing it.

    Sent from me!  Please pardon any tpyos


    On May 17, 2014, at 1:18 AM, Blair Trosper
    <blair.trosper@xxxxxxxxxxxxxxxxxxx
    <mailto:blair.trosper@xxxxxxxxxxxxxxxxxxx>> wrote:

    No joy there.  Our providers (TAMU, OU, WISC, et al) are all
    under tier 1 providers, and there is no lag.

    I can't find the document that describes how the NEXRAD2 data is
    fed into the LDM system.  I think it happens in DC at the
    Internet2 aperture, but that information may be out of date (or I
    may be remembering it wrong).

    As I recall, two weeks ago, we weren't the only ones seeing this lag.

    And we're still seeing it on three servers with three separate
    upstream providers -- and each server is deliberately on
    different BGP/backbone providers for geographic AND network
    diversity.

    Something's not quite right, and I think the key is understanding
    how the NEXRAD2 data gets into LDM...and then going to THAT
    source to find out why it's delayed.  I know Allison House's data
    isn't delayed (whereas iastate's is), so my guess is they're
    acquiring it independently of LDM.

    So we know iastate.edu <http://iastate.edu> is getting data 10
    minutes late just like we are, as are our upstream
    providers...but the servers themselves are always at 0 or 1
    second of lag.

    Something is definitely wrong.  We don't see any lag on any of
    the other feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3,
    etc).  This seems to be unique to NEXRAD2.

    Can someone from UCAR/UNIDATA jump in and help us...or should we
    open a formal support ticket?


    On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson
    <emmerson.steven@xxxxxxxxx <mailto:emmerson.steven@xxxxxxxxx>> wrote:

        It could be that the last NEXRAD2 data-product is stuck in a
        system buffer. Try putting a "-flush" or "-close" on the
        relevant PIPE action.

        On May 11, 2014 11:49 AM, "Gilbert Sebenste"
        <sebenste@xxxxxxxxxxxxxxxxxxxxx
        <mailto:sebenste@xxxxxxxxxxxxxxxxxxxxx>> wrote:

            Hi Blair,

                We are still seeing this.  I know the TOC was made
                aware of this issue two
                weeks ago, but despite all our servers being 0
                seconds lagged on the
                NEXRAD2 feed tree, our data (from multiple providers
                who are also not
                lagged) is at least 10 minutes behind.


            Been off the grid, for the most part, or I would have
            answered earlier. Things to look for:

            Can you do a traceroute to the servers in question? See
            if anything is
            jammed up inbetween them and you.

            Are all network cards set at least at 100 mb? If 10 mb,
            it can't handle it.

            Is the system clock correct? Are you using ntp? This is a
            must.

                Also, this page is a 500 internal server error and
                has been for months, and nobody has fixed it or seems
                to even notice:
                
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_topogif?NEXRAD2

                And this one takes over a minute to load:
                
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_feedtree?NEXRAD2

                ...but you can verify from the second link that we're
                not lagging (search
                for "tampa" or "chicago").


            Give those a try aboove, and see what happens.

            Gilbert

            
*******************************************************************************
            Gilbert Sebenste  ********
            (My opinions only!)  ******
            Staff Meteorologist, Northern Illinois University      ****
            E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx
<mailto:sebenste@xxxxxxxxxxxxxxxxxxxxx> ***
            web: http://weather.admin.niu.edu  **
Twitter: http://www.twitter.com/NIU_Weather ** Facebook: http://www.facebook.com/niu.weather *
            
*******************************************************************************

            _______________________________________________
            ldm-users mailing list
            ldm-users@xxxxxxxxxxxxxxxx
            <mailto:ldm-users@xxxxxxxxxxxxxxxx>
            For list information or to unsubscribe,  visit:
            http://www.unidata.ucar.edu/mailing_lists/


        _______________________________________________
        ldm-users mailing list
        ldm-users@xxxxxxxxxxxxxxxx <mailto:ldm-users@xxxxxxxxxxxxxxxx>
        For list information or to unsubscribe,  visit:
        http://www.unidata.ucar.edu/mailing_lists/




--
    ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
    ?? ?? ?? ?? ?? ?? ?? ?? ?
    Blair Trosper
    Updraft Networks / Weather Data
      NOC: 469-844-5440 <tel:469-844-5440>
      Early Watch Notifications: http://twitter.com/weatherwatches
    _______________________________________________
    ldm-users mailing list
    ldm-users@xxxxxxxxxxxxxxxx <mailto:ldm-users@xxxxxxxxxxxxxxxx>
    For list information or to unsubscribe,  visit:
    http://www.unidata.ucar.edu/mailing_lists/




--

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ?? ?? ?? ?? ?? ?? ?? ?
Blair Trosper
  Updraft Networks / Weather Data
  NOC:  469-844-5440
  Early Watch Notifications: http://twitter.com/weatherwatches


_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/

  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: