[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: hung downstream LDM

Justin,

>Date: Mon, 31 Oct 2005 10:40:58 -0500
>From: Justin Cooke <address@hidden>
>Organization: NOAA
>To: Steve Emmerson <address@hidden>
>Subject: Re: hung downstream LDM

The above message contained the following:

[snip]
> TOC got the log files to me.
> 
> When our feed ended:
> 
> Oct 28 07:46:36 tgp49 tgp49[25892] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 07:47:39 tgp49 tgp49[26605] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 07:48:49 tgp49 tgp49[27297] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 07:50:09 tgp49 tgp49[27999] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 07:51:27 tgp49 tgp49[29383] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 07:52:34 tgp49 tgp49[30099] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 07:53:37 tgp49 tgp49[30782] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 07:54:52 tgp49 tgp49[31470] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 07:56:13 tgp49 tgp49[32169] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 07:57:17 tgp49 tgp49[391] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 07:58:23 tgp49 tgp49[1444] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 07:59:33 tgp49 tgp49[2527] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 08:00:50 tgp49 tgp49[3227] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 08:02:01 tgp49 tgp49[3918] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 08:03:23 tgp49 tgp49[5065] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> 
> 
> When I restarted our LDM (192.58.3.196)
> 
> Oct 28 11:03:37 tgp49 tgp49[951] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:04:52 tgp49 tgp49[1640] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:06:16 tgp49 tgp49[2360] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:07:17 tgp49 tgp49[3045] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:08:27 tgp49 tgp49[4200] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:09:44 tgp49 tgp49[5130] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:10:29 tgp49 192.58.3.196(feed)[22240] ERROR: feed or notify 
> failure; HEREIS: RPC: Unable to send; errno = Broken pipe 
> Oct 28 11:10:30 tgp49 rpc.ldmd[21917] NOTE: child 22240 exited with status 7 
> Oct 28 11:10:54 tgp49 192.58.3.196(feed)[5831] NOTE: Starting Up(6.4.2/6): 
> 20051028101053.061 TS_ENDT {{NEXRAD2,  ".*"}}, Primary 
> Oct 28 11:10:54 tgp49 192.58.3.196(feed)[5831] NOTE: topo:  192.58.3.196 
> {{NEXRAD2, (.*)}} 
> Oct 28 11:11:07 tgp49 tgp49[5832] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:12:37 tgp49 tgp49[7230] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:13:46 tgp49 tgp49[7920] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:14:49 tgp49 tgp49[8607] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:16:08 tgp49 tgp49[9308] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  ".*"}} 
> Oct 28 11:17:29 tgp49 tgp49[10713] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> Oct 28 11:18:51 tgp49 tgp49[11405] NOTE: hiya6: TS_ZERO TS_ENDT {{EXP,  
> ".*"}} 
> let me know if you would like the complete log file from 10/28....but to 
> me nothing stood out.

That's OK, I don't need the complete log file.  You're right, nothing
stands out.

I think you were also right in your suspicion that something was wrong
with logging and, conversely, I think I was mistaken in my original
diagnosis that the problem lay with the upstream LDM.

I'll need to verify and validate the new logging module on our AIX
system -- as well as insure that it doesn't break on any of the other
operating-systems.  Also, my daughter is returning for New York City
today and I've been too ill to spend much time with her.  Consequently,
I probably won't have a release for you until tomorrow -- for which I
apologize.

In the meantime, try to keep the logging by your downstream,
NEXRAD2-receiving LDM to a minimum in order to reduce the chances of
hanging.

Regards,
Steve Emmerson