Re: [ldm-users] Level 2 radar issues from NWS CRH

Okay, here's what I've been able to dig up.  It looks like there's a delay of 
30 minutes or more in _some_ CR products.  A few seconds of ldmadmin watch 
turned up these (I ignored anything that was less than 10 minutes late):

Jul 04 14:54:13 pqutil INFO:    61365 20090704142559.640 NEXRAD2 558018  
L2-BZIP2/KSGF/20090704142506/558/18/I/V03/0
Jul 04 14:54:14 pqutil INFO:    15786 20090704142559.931 NEXRAD2 84028  
L2-BZIP2/KLNX/20090704141924/84/28/I/V03/0
Jul 04 14:54:20 pqutil INFO:    48059 20090704142600.113 NEXRAD2 586004  
L2-BZIP2/KDMX/20090704142551/586/4/I/V03/0
Jul 04 14:54:21 pqutil INFO:    55683 20090704142600.513 NEXRAD2 697049  
L2-BZIP2/KDVN/20090704142326/697/49/I/V03/0
Jul 04 14:54:27 pqutil INFO:     8958 20090704142602.097 NEXRAD2 541063  
L2-BZIP2/KOAX/20090704142141/541/63/I/V03/0
Jul 04 14:54:32 pqutil INFO:    23970 20090704142602.264 NEXRAD2 929039  
L2-BZIP2/KICT/20090704142354/929/39/I/V03/0
Jul 04 14:54:35 pqutil INFO:    60772 20090704142600.912 NEXRAD2 647018  
L2-BZIP2/KLSX/20090704142507/647/18/I/V03/0
Jul 04 14:54:37 pqutil INFO:    68477 20090704142602.760 NEXRAD2 414046  
L2-BZIP2/KILX/20090704142320/414/46/I/V03/0
Jul 04 14:54:39 pqutil INFO:    31960 20090704142603.111 NEXRAD2 768023  
L2-BZIP2/KDLH/20090704142124/768/23/I/V03/0
Jul 04 14:54:51 pqutil INFO:    21836 20090704142617.992 NEXRAD2 659003  
L2-BZIP2/KMQT/20090704142554/659/3/I/V03/0
Jul 04 14:54:52 pqutil INFO:    43730 20090704142618.445 NEXRAD2 281057  
L2-BZIP2/KMPX/20090704142317/281/57/I/V03/0
Jul 04 14:54:53 pqutil INFO:    27219 20090704142618.479 NEXRAD2 311038  
L2-BZIP2/KARX/20090704142314/311/38/I/V04/0
Jul 04 14:54:53 pqutil INFO:    22742 20090704142618.867 NEXRAD2 697053  
L2-BZIP2/KDVN/20090704142326/697/53/I/V03/0

So I went back and looked at file write times.  It looks like CR sites started 
seeing a delay (generally ~30 min) around 4Z on 2 July.  This was fairly 
consistent in the sites I looked at.  Sites in other regions did not exhibit 
the same latency.  (Time stamps are in UTC-4)

-rw-r--r-- 1 ldm ldm 1130776 Jul  1 23:32 KDMX_20090702_0322
-rw-r--r-- 1 ldm ldm 1152867 Jul  1 23:42 KDMX_20090702_0332
-rw-r--r-- 1 ldm ldm 1163772 Jul  1 23:51 KDMX_20090702_0342
-rw-r--r-- 1 ldm ldm 1864256 Jul  2 00:15 KDMX_20090702_0351
-rw-r--r-- 1 ldm ldm 2328948 Jul  2 00:40 KDMX_20090702_0401
-rw-r--r-- 1 ldm ldm 2323290 Jul  2 00:57 KDMX_20090702_0411
-rw-r--r-- 1 ldm ldm 1752044 Jul  2 01:08 KDMX_20090702_0421
-rw-r--r-- 1 ldm ldm 1865551 Jul  2 01:17 KDMX_20090702_0431
-rw-r--r-- 1 ldm ldm 1817208 Jul  2 01:27 KDMX_20090702_0441

-rw-r--r-- 1 ldm ldm  984866 Jul  1 23:25 KILX_20090702_0315
-rw-r--r-- 1 ldm ldm  979181 Jul  1 23:34 KILX_20090702_0325
-rw-r--r-- 1 ldm ldm  977704 Jul  1 23:44 KILX_20090702_0334
-rw-r--r-- 1 ldm ldm  975361 Jul  1 23:54 KILX_20090702_0344
-rw-r--r-- 1 ldm ldm 1946615 Jul  2 00:25 KILX_20090702_0354
-rw-r--r-- 1 ldm ldm 1958218 Jul  2 00:44 KILX_20090702_0403
-rw-r--r-- 1 ldm ldm 1918879 Jul  2 00:58 KILX_20090702_0413
-rw-r--r-- 1 ldm ldm 1484241 Jul  2 01:08 KILX_20090702_0423
-rw-r--r-- 1 ldm ldm 1546451 Jul  2 01:17 KILX_20090702_0432
-rw-r--r-- 1 ldm ldm 1700754 Jul  2 01:27 KILX_20090702_0442
-rw-r--r-- 1 ldm ldm 1650739 Jul  2 01:38 KILX_20090702_0452
-rw-r--r-- 1 ldm ldm 1451028 Jul  2 01:47 KILX_20090702_0501

-rw-r--r-- 1 ldm ldm 1903809 Jul  1 23:38 KOAX_20090702_0334
-rw-r--r-- 1 ldm ldm 1935148 Jul  1 23:43 KOAX_20090702_0338
-rw-r--r-- 1 ldm ldm 1948071 Jul  1 23:48 KOAX_20090702_0343
-rw-r--r-- 1 ldm ldm 1940662 Jul  1 23:52 KOAX_20090702_0348
-rw-r--r-- 1 ldm ldm 2830923 Jul  2 00:08 KOAX_20090702_0352
-rw-r--r-- 1 ldm ldm 3889692 Jul  2 00:15 KOAX_20090702_0357
-rw-r--r-- 1 ldm ldm 3909672 Jul  2 00:30 KOAX_20090702_0402
-rw-r--r-- 1 ldm ldm 3953484 Jul  2 00:40 KOAX_20090702_0406
-rw-r--r-- 1 ldm ldm 2449726 Jul  2 00:52 KOAX_20090702_0411
-rw-r--r-- 1 ldm ldm 2403103 Jul  2 00:59 KOAX_20090702_0417
-rw-r--r-- 1 ldm ldm 2155576 Jul  2 01:06 KOAX_20090702_0424
-rw-r--r-- 1 ldm ldm 1971551 Jul  2 01:13 KOAX_20090702_0430

Here's current data.  

-rw-r--r-- 1 ldm ldm 2769972 Jul  4 11:40 KDMX_20090704_1507
-rw-r--r-- 1 ldm ldm 2762001 Jul  4 11:45 KDMX_20090704_1512
-rw-r--r-- 1 ldm ldm 2731136 Jul  4 11:48 KDMX_20090704_1517
-rw-r--r-- 1 ldm ldm 2344602 Jul  4 11:26 KDMX_20090704_1521
-rw-r--r-- 1 ldm ldm 2343846 Jul  4 11:31 KDMX_20090704_1526
-rw-r--r-- 1 ldm ldm 2314881 Jul  4 11:36 KDMX_20090704_1531
-rw-r--r-- 1 ldm ldm 2280884 Jul  4 11:41 KDMX_20090704_1536
-rw-r--r-- 1 ldm ldm 2280884 Jul  4 11:46 KDMX_20090704_1541
-rw-r--r-- 1 ldm ldm 1755661 Jul  4 11:48 KDMX_20090704_1546

-rw-r--r-- 1 ldm ldm 6049570 Jul  4 11:43 KILX_20090704_1509
-rw-r--r-- 1 ldm ldm 5955236 Jul  4 11:47 KILX_20090704_1514
-rw-r--r-- 1 ldm ldm 5823187 Jul  4 11:49 KILX_20090704_1519
-rw-r--r-- 1 ldm ldm 5616058 Jul  4 11:28 KILX_20090704_1523
-rw-r--r-- 1 ldm ldm 5597159 Jul  4 11:33 KILX_20090704_1528
-rw-r--r-- 1 ldm ldm 5592560 Jul  4 11:37 KILX_20090704_1533
-rw-r--r-- 1 ldm ldm 5589902 Jul  4 11:43 KILX_20090704_1537
-rw-r--r-- 1 ldm ldm 5568192 Jul  4 11:47 KILX_20090704_1543

-rw-r--r-- 1 ldm ldm 1021825 Jul  4 11:06 KOAX_20090704_1432
-rw-r--r-- 1 ldm ldm  904902 Jul  4 11:11 KOAX_20090704_1437
-rw-r--r-- 1 ldm ldm  927664 Jul  4 11:17 KOAX_20090704_1442
-rw-r--r-- 1 ldm ldm  553907 Jul  4 11:27 KOAX_20090704_1448
-rw-r--r-- 1 ldm ldm  556596 Jul  4 11:35 KOAX_20090704_1458
-rw-r--r-- 1 ldm ldm  549581 Jul  4 11:48 KOAX_20090704_1509
-rw-r--r-- 1 ldm ldm  543498 Jul  4 11:49 KOAX_20090704_1519
-rw-r--r-- 1 ldm ldm  512042 Jul  4 11:40 KOAX_20090704_1530

I've checked my LDM box for sanity, and it appears to be sane.  I wonder 
though... it looks like castor.itap.purdue.edu has MUCH higher latency than 
you're seeing, so perhaps it's a problem with our top-tier site?  Where are the 
others who are seeing this feeding from?  I've bothered the admin for 
castor.itap.purdue.edu and pollux.itap.purdue.edu via his roommate on Facebook, 
so perhaps he'll weigh in later this afernoon.  I'm told he's already seen our 
messages and is looking into the problem.


--
Ben Cotton
Unix System Administrator
Department of Earth and
  Atmospheric Sciences
Purdue University
http://www.purdue.edu/eas/resources/it

CIVL 4212
O: 765 49-61546
C: 765 404-7986



-----Original Message-----
From: Gerry Creager [mailto:gerry.creager@xxxxxxxx]
Sent: Sat 7/4/2009 9:03 AM
To: Cotton, Benjamin J
Cc: sebenste@xxxxxxxxxxxxxxxxxxxxx; ldm-users@xxxxxxxxxxxxxxxx
Subject: Re: [ldm-users] Level 2 radar issues from NWS CRH
 
I've checked my data cache, and I don't see any "interesting" latencies 
on those three sites.  I suppose I could set up a script to do a better 
job of monitoring things like that, though.

Last-write times were consistent with file-size for clear-air mode, and 
similarly, for normal "storm" mode.  Until I get more coffee, I'm not 
likely to decode the files to look for when folks changed to a 
storm-related VCP.

I had about an 800 sec spike in Level II latencies, as seen at 
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD2+bigbird.tamu.edu
about or shortly after 03/00z

gerry

Cotton, Benjamin J wrote:
> I specifically remember seeing delays for KICT, KDMX, and KLOT.  That 
> was only from a quick peek and I haven't looked again since this 
> morning.  I'll take a deeper look Saturday morning.
> 
> --
> (Sent via Blackberry).
> Ben Cotton, KC9FYX
> Unix System Administrator
> Department of Earth and
>   Atmospheric Sciences
> Purdue University
> http://www.purdue.edu/eas/resources/it
> 
> CIVL 4212
> O: 765 49-61546
> C: 765 404-7986
> 
> ----- Original Message -----
> From: ldm-users-bounces@xxxxxxxxxxxxxxxx 
> <ldm-users-bounces@xxxxxxxxxxxxxxxx>
> To: Gilbert Sebenste <sebenste@xxxxxxxxxxxxxxxxxxxxx>
> Cc: ldm-users@xxxxxxxxxxxxxxxx <ldm-users@xxxxxxxxxxxxxxxx>
> Sent: Sat Jul 04 00:07:48 2009
> Subject: Re: [ldm-users] Level 2 radar issues from NWS CRH
> 
> I've not done an exhaustive search, so this isn't significant from a
> sampling viewpoint, but the several sites I've looked at from CR didn't
> show a lot of latency.  I recalled the CRH maintenance, like Gilbert
> did, and thought that could be playing some roll, but I'm not sure.
> 
> What sites are folks seeing a problem drawing from?
> 
> gerry
> 
> Gilbert Sebenste wrote:
>  > FYI...could this be somewhere in the UNIDATA chain?
>  >
>  > 
> *******************************************************************************
>  >
>  > Gilbert Sebenste                                                    
>  > ********
>  > (My opinions only!)                                                  
> ******
>  > Staff Meteorologist, Northern Illinois 
> University                      ****
>  > E-mail: 
> sebenste@xxxxxxxxxxxxxxxxxxxxx                                  ***
>  > web: http://weather.admin.niu.edu                                      **
>  > 
> *******************************************************************************
>  >
>  >
>  > ---------- Forwarded message ----------
>  > Date: Fri, 03 Jul 2009 17:59:55 -0500
>  > From: Tim.D.Crum@xxxxxxxx
>  > To: Gilbert Sebenste <sebenste@xxxxxxxxxxxxxxxxxxxxx>
>  > Subject: Re: Level 2 radar issues from NWS CRH
>  >
>  > Gilbert,
>  >
>  > Please tell me more.  The ERC, NWS and OU web sites show low Central
>  > Region latencies.  I spoke with ERC and they are not seeing duplicate
>  > copies of Central Region Level II data.
>  >
>  >
>  > Tim
>  >
>  >
>  > ----- Original Message -----
>  > From: Gilbert Sebenste <sebenste@xxxxxxxxxxxxxxxxxxxxx>
>  > Date: Friday, July 3, 2009 1:40 pm
>  > Subject: Level 2 radar issues from NWS CRH
>  >> Hi Tim,
>  >>
>  >> Level2 radar data from Central Region appears to be going out in
>  >> duplicate, with the second copy of the file delayed by up to an
>  >> hour.
>  >> This is only happening with NWS Central Region sites; other
>  >> regions are
>  >> fine. Can those delayed duplicates be shut off? This started with
>  >> network
>  >> maintenance last night at CR; they said they were going to do work
>  >> from 9
>  >> PM to midnight, and as they announced it was completed, that's
>  >> when the
>  >> trouble started.
>  >>
>  >>
>  > ***********************************************************************
>  > ********
>  >> Gilbert Sebenste
>  >>  ********
>  >> (My opinions only!)
>  >>  ******
>  >> Staff Meteorologist, Northern Illinois University
>  >>    ****
>  >> E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx
>  >>     ***
>  >> web: http://weather.admin.niu.edu
>  >>    **
>  >>
>  > ***********************************************************************
>  > ********
>  >>
>  >
>  > _______________________________________________
>  > ldm-users mailing list
>  > ldm-users@xxxxxxxxxxxxxxxx
>  > For list information or to unsubscribe,  visit:
>  > http://www.unidata.ucar.edu/mailing_lists/
> 
> --
> Gerry Creager -- gerry.creager@xxxxxxxx
> Texas Mesonet -- AATLT, Texas A&M University   
> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
> Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
> 
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/
> 

-- 
Gerry Creager -- gerry.creager@xxxxxxxx
Texas Mesonet -- AATLT, Texas A&M University    
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843

  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: