[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20030823: zoneWriter run amok on motherlode on Saturday morning

>From: Unidata User Support <address@hidden>
>Organization: Unidata Program Center/UCAR
>Keywords: 200308231227.h7NCR2Ld027291 IDD LDM zoneWriter FNEXRAD

Guys,

Saturday morning we received an email from Jim Koermer that let us
know that the NEXRAD 1 km National composite base reflectivity images
were not being relayed by atm:

  >From address@hidden Sat Aug 23 06:27:03 2003
  
  >Hello,
  
  >I think that there is a problem with atm.geo.nsf.gov, but only for the 
  >1KN0R-NAT data. Other composites have been updating.
  
  >Jim
  >--
  >James P. Koermer             E-Mail: address@hidden
  >Professor of Meteorology     Office Phone: (603)535-2574
  >Natural Science Department   Office Fax: (603)535-2723
  >Plymouth State College       WWW: http://vortex.plymouth.edu/
  >Plymouth, NH 03264

I verified that the 1 km national composites were not available (through
ADDE) on either atm or motherlode.  When I logged onto motherlode, I
found that the load average was well over 90:

-- snippit from ~ldm/logs/motherlode.uptime --

 ...
20030823.1518  99.95 96.07 94.69    2  14  16   3933
20030823.1519  99.43 96.68 95.00    2  14  16   3988
20030823.1520 101.09 97.73 95.49    2  14  16   3996
20030823.1521 100.08 97.94 95.70    2  14  16   4023
20030823.1522  99.79 98.30 95.98    2  15  17   4045
20030823.1523  97.66 98.00 96.03    2  14  16   4016
20030823.1524  95.93 97.51 95.98    2  14  16   4020
20030823.1525  97.48 97.63 96.12    2  14  16   3979
20030823.1526  97.23 97.51 96.17    2  14  16   3990
20030823.1527  94.97 96.90 96.05    2  14  16   3377
20030823.1528  97.33 97.16 96.20    2  15  17   3442
20030823.1529  99.89 97.93 96.53    2  14  16   3498
20030823.1530 100.40 98.48 96.82    2  14  16   3560
20030823.1531 100.54 98.88 97.07    2  14  16   3617
20030823.1532  99.62 98.93 97.21    2  14  16   3680
 ...


The offending culprit was a hundred (estimate) invocations of
~ldm/decoders/zoneWriter.  Not knowing who's decoder this is (Robb?,
zoneWriter is a Perl script), I decided to send this note to a list of
possible candidates.

In order to get things working again, I stopped the LDM to kill all of
the zoneWriter invocations and commented its execution out of the
pqact.conf file.  As I write this, things are slowly returning to
normal, or, at least, some new composites are being sent out in the
FNEXRAD feed.

Tom