Ryan has informed me that I've gotten old and can't remember what I've had for
breakfast. We run ext4 but the options have been tuned making it "satisfactory".
-Tyler
> On May 23, 2014, at 4:39 PM, Gerry Creager - NOAA Affiliate
> <gerry.creager@xxxxxxxx> wrote:
>
> I'll have to take an alternate stance to both RAID5/6 and ext4. Our
> experience with ext4 was not as good as ext2, but more than satisfactory.
> However, xfs was the order of the day when I was still at A&M. Similarly, the
> difference between RAID0 and RAID5 is noticable, but I had better than "just
> satisfactory" performance using RAID6, and fewer worries about hardware
> failures. I'm suspecting a time issue, offhand.
>
> gerry
>
>
>> On Fri, May 23, 2014 at 1:13 PM, Robert Mullenax
>> <Robert.Mullenax@xxxxxxxxxxxxx> wrote:
>> At this juncture it sure sounds like it almost has to be a disk performance
>> issue. I went from HW RAID 0 to HW RAID 5 a few years ago as a test and was
>> shocked to see how much slower writes are..not that I am advocating RAID 0.
>> We use ZFS mirroring now.
>>
>>
>>
>>
>> -----Original Message-----
>> From: ldm-users-bounces@xxxxxxxxxxxxxxxx on behalf of Tyler Allison
>> Sent: Fri 5/23/2014 12:53 PM
>> To: Ryan Hickman
>> Cc: LDM Users
>> Subject: Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
>>
>> I think you said you're running ext4 files system on the data directory?
>>
>> My testing of ext4 with high IO LDM servers was atrocious. We no longer use
>> it for our high speed disk write areas.
>>
>> If you can, mount an ext2 partition and move your writes to there for a test
>> .
>>
>> -Tyler
>>
>> > On May 23, 2014, at 12:34 PM, Ryan Hickman <ryan@xxxxxxxxxxxxxxxx> wrote:
>> >
>> > Let's see your NEXRAD2 pqact entries, please. We also didn't see the
>> > relevant 'ldmadmin watch' that Daryl suggested either so we could
>> > determine LDM product queue insertion time.
>> >
>> > This doesn't sound like a network topology or product dissemination issue;
>> > other list members would likely be seeing this problem, too. However, I'd
>> > be willing to add an "allow" for NEXRAD2 from AllisonHouse's Texas servers
>> > for 48 hours just to help you rule that out. Feel free to email me off
>> > list to accomplish this.
>> >
>> > Otherwise I don't have many other ideas.
>> >
>> > P.S. Daryl Herzmann has been trying to help. But it seems as if you aren't
>> > receiving his messages.
>> > http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00059.html
>> >
>> > Daryl - I noticed that you might be manually removing the email quotes in
>> > your replies. This MIGHT be throwing the Mailing List software off. If you
>> > trace the "Follow Ups" on Blair's original message
>> > (http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00044.html),
>> > I never see your replies. If you look at Blair's quoted messages, you'll
>> > see that his replies only quote the "Follow Ups" and they never include
>> > your replies.
>> >
>> > Blair - Consider adding Daryl to your email contacts. Or check your Spam /
>> > Junk folders for his previous messages.
>> >
>> >
>> >> On Thu, May 22, 2014 at 8:29 PM, Blair Trosper
>> >> <blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>> >> We're still getting it from all of our servers, and I added another one
>> >> in Denver with new providers again, and still seeing the same issue.
>> >> Unique to NEXRAD2.
>> >>
>> >> Can anyone volunteer a server to feed us NEXRAD2 data for a few days (or
>> >> as a backup permanently) to try and troubleshoot this?
>> >>
>> >> None of our servers is on the same network, so it cannot be that. And
>> >> these are all rather powerful machines with extremely high connectivity
>> >> and throughput...and we don't see latency on any of the other feeds we
>> >> carry.
>> >>
>> >> We're only utilizing 1-2% of our total throughput on our Chicago server,
>> >> for example, but we're getting every site 10 minutes delayed from:
>> >> - OU
>> >> - TAMU
>> >> - WISC
>> >>
>> >> ...but none of us four shows any lag in the topology.
>> >>
>> >> What else could be going on?
>> >>
>> >>
>> >>> On Sat, May 17, 2014 at 9:35 AM, Karen Cooper - NOAA Affiliate
>> >>> <karen.cooper@xxxxxxxx> wrote:
>> >>> The level 2 data is inserted into Ldm queues at the radar RPGs as it is
>> >>> generated. If it were a problem there everyone would be seeing it.
>> >>>
>> >>> Sent from me! Please pardon any tpyos
>> >>>
>> >>>
>> >>>> On May 17, 2014, at 1:18 AM, Blair Trosper
>> >>>> <blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>> >>>>
>> >>>> No joy there. Our providers (TAMU, OU, WISC, et al) are all under tier
>> >>>> 1 providers, and there is no lag.
>> >>>>
>> >>>> I can't find the document that describes how the NEXRAD2 data is fed
>> >>>> into the LDM system. I think it happens in DC at the Internet2
>> >>>> aperture, but that information may be out of date (or I may be
>> >>>> remembering it wrong).
>> >>>>
>> >>>> As I recall, two weeks ago, we weren't the only ones seeing this lag .
>> >>>>
>> >>>> And we're still seeing it on three servers with three separate upstream
>> >>>> providers -- and each server is deliberately on different BGP/backbone
>> >>>> providers for geographic AND network diversity.
>> >>>>
>> >>>> Something's not quite right, and I think the key is understanding how
>> >>>> the NEXRAD2 data gets into LDM...and then going to THAT source to find
>> >>>> out why it's delayed. I know Allison House's data isn't delayed
>> >>>> (whereas iastate's is), so my guess is they're acquiring it
>> >>>> independently of LDM.
>> >>>>
>> >>>> So we know iastate.edu is getting data 10 minutes late just like we
>> >>>> are, as are our upstream providers...but the servers themselves are
>> >>>> always at 0 or 1 second of lag.
>> >>>>
>> >>>> Something is definitely wrong. We don't see any lag on any of the
>> >>>> other feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3, etc). This
>> >>>> seems to be unique to NEXRAD2.
>> >>>>
>> >>>> Can someone from UCAR/UNIDATA jump in and help us...or should we open a
>> >>>> formal support ticket?
>> >>>>
>> >>>>
>> >>>>> On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson
>> >>>>> <emmerson.steven@xxxxxxxxx> wrote:
>> >>>>> It could be that the last NEXRAD2 data-product is stuck in a system
>> >>>>> buffer. Try putting a "-flush" or "-close" on the relevant PIPE action.
>> >>>>>
>> >>>>>> On May 11, 2014 11:49 AM, "Gilbert Sebenste"
>> >>>>>> <sebenste@xxxxxxxxxxxxxxxxxxxxx> wrote:
>> >>>>>> Hi Blair,
>> >>>>>>
>> >>>>>>> We are still seeing this. I know the TOC was made aware of this
>> >>>>>>> issue two
>> >>>>>>> weeks ago, but despite all our servers being 0 seconds lagged on the
>> >>>>>>> NEXRAD2 feed tree, our data (from multiple providers who are also not
>> >>>>>>> lagged) is at least 10 minutes behind.
>> >>>>>>
>> >>>>>> Been off the grid, for the most part, or I would have answered
>> >>>>>> earlier. Things to look for:
>> >>>>>>
>> >>>>>> Can you do a traceroute to the servers in question? See if anything is
>> >>>>>> jammed up inbetween them and you.
>> >>>>>>
>> >>>>>> Are all network cards set at least at 100 mb? If 10 mb, it can't
>> >>>>>> handle it.
>> >>>>>>
>> >>>>>> Is the system clock correct? Are you using ntp? This is a must.
>> >>>>>>
>> >>>>>>> Also, this page is a 500 internal server error and has been for
>> >>>>>>> months, and nobody has fixed it or seems to even notice:
>> >>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_topogif?NEXRAD2
>> >>>>>>>
>> >>>>>>> And this one takes over a minute to load:
>> >>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_feedtree?NEXRAD2
>> >>>>>>>
>> >>>>>>> ...but you can verify from the second link that we're not lagging
>> >>>>>>> (search
>> >>>>>>> for "tampa" or "chicago").
>> >>>>>>
>> >>>>>> Give those a try aboove, and see what happens.
>> >>>>>>
>> >>>>>> Gilbert
>> >>>>>>
>> >>>>>> *******************************************************************************
>> >>>>>> Gilbert Sebenste
>> >>>>>> ********
>> >>>>>> (My opinions only!)
>> >>>>>> ******
>> >>>>>> Staff Meteorologist, Northern Illinois University
>> >>>>>> ****
>> >>>>>> E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx
>> >>>>>> ***
>> >>>>>> web: http://weather.admin.niu.edu
>> >>>>>> **
>> >>>>>> Twitter: http://www.twitter.com/NIU_Weather
>> >>>>>> **
>> >>>>>> Facebook: http://www.facebook.com/niu.weather
>> >>>>>> *
>> >>>>>> *******************************************************************************
>> >>>>>>
>> >>>>>> _______________________________________________
>> >>>>>> ldm-users mailing list
>> >>>>>> ldm-users@xxxxxxxxxxxxxxxx
>> >>>>>> For list information or to unsubscribe, visit:
>> >>>>>> http://www.unidata.ucar.edu/mailing_lists/
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> ldm-users mailing list
>> >>>>> ldm-users@xxxxxxxxxxxxxxxx
>> >>>>> For list information or to unsubscribe, visit:
>> >>>>> http://www.unidata.ucar.edu/mailing_lists/
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>>
>> >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>> >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>> >>>> ?
>>
>> >>>> Blair Trosper
>> >>>> Updraft Networks / Weather Data
>> >>>> NOC: 469-844-5440
>> >>>> Early Watch Notifications: http://twitter.com/weatherwatches
>> >>>> _______________________________________________
>> >>>> ldm-users mailing list
>> >>>> ldm-users@xxxxxxxxxxxxxxxx
>> >>>> For list information or to unsubscribe, visit:
>> >>>> http://www.unidata.ucar.edu/mailing_lists/
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>>
>> >> Blair Trosper
>> >> Updraft Networks / Weather Data
>> >> NOC: 469-844-5440
>> >> Early Watch Notifications: http://twitter.com/weatherwatches
>> >>
>> >> _______________________________________________
>> >> ldm-users mailing list
>> >> ldm-users@xxxxxxxxxxxxxxxx
>> >> For list information or to unsubscribe, visit:
>> >> http://www.unidata.ucar.edu/mailing_lists/
>> >
>> > _______________________________________________
>> > ldm-users mailing list
>> > ldm-users@xxxxxxxxxxxxxxxx
>> > For list information or to unsubscribe, visit:
>> > http://www.unidata.ucar.edu/mailing_lists/
>>
>>
>> _______________________________________________
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
>
>
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371
> ++++++++++++++++++++++
> “Big whorls have little whorls,
> That feed on their velocity;
> And little whorls have lesser whorls,
> And so on to viscosity.”
> Lewis Fry Richardson (1881-1953)
