[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #MKY-400850]: Continuing issue with no TIRC headers getting into secondary LDM box

Gilbert,

Now we're getting somewhere.

The "Unable to receive" message are due to an upstream LDM process on Bird01 
that's sending notifications to a notifyme(1) process on Ldm01 not getting a 
response to a notification from the notifyme(1) process within 35 seconds 
(which is a very long time). Consequently, there appears to be some impediment 
preventing or delaying responses from a notifyme(1) on Ldm01 from reaching 
Bird01. This could be due to a miss-configured firewall (e.g., iptables), or a 
network monitor throttling the connection, or a bad Internet connection, etc.

It could also be due to killing the notifyme(1) process on Ldm01. I suggest 
running a notifyme(1) process on Ldm01 while watching the LDM log file on 
Bird01 to see if the corresponding upstream LDM exits before the notifyme(1) is 
terminated.

> Just tried a NOTIFYME on ldm01 to bird01, and this is what I got in
> /home/ldm/var/log/ldmd.log:
> 
> 20170830T220608.936370Z 104.197.198.101(noti)[28678] NOTE
> forn5_svc.c:468:forn_5_svc() Starting Up(6.13.6/5): 20170830220608.883453
> TS_ENDT {{NOTHER, "TIRC"}}
> 20170830T220608.936427Z 104.197.198.101(noti)[28678] NOTE
> forn5_svc.c:471:forn_5_svc() topo:  104.197.198.101 NOTHER
> 
> Now, it has started to show TIRC data again, but it is intermittent. Most
> of the day, we get nothing.
> 
> From the day before, i saw this, however:
> 
> 20170828T233516.872211Z 127.0.0.1(noti)[15057] NOTE
> forn5_svc.c:468:forn_5_svc() Starting Up(6.13.6/5): 20170828233516.869028
> TS_ENDT {{NOTHER, "TIRC"}}
> 20170828T233516.872272Z 127.0.0.1(noti)[15057] NOTE
> forn5_svc.c:471:forn_5_svc() topo:  127.0.0.1 NOTHER
> 20170828T233523.745069Z 127.0.0.1(noti)[15057] ERROR
> forn5_svc.c:273:noti5_sqf() TIRC15 KNES 282332 PAE: RPC: Unable to receive
> 20170828T233523.745166Z 127.0.0.1(noti)[15057] ERROR
> forn5_svc.c:554:forn_5_svc() pq_sequence failed: Input/output error (errno
> = 5)
> 20170828T233523.745222Z 127.0.0.1(noti)[15057] NOTE ldmd.c:187:cleanup()
> Exiting
> 20170828T233523.746139Z ldmd[2785] NOTE ldmd.c:170:reap() child 15057
> exited with status 1
> 20170828T233527.478743Z 104.197.198.101(noti)[14906] ERROR
> forn5_svc.c:273:noti5_sqf() TIRC05 KNES 282332 PAU: RPC: Unable to receive
> 20170828T233527.478799Z 104.197.198.101(noti)[14906] ERROR
> forn5_svc.c:554:forn_5_svc() pq_sequence failed: Input/output error (errno
> = 5)
> 20170828T233527.478825Z 104.197.198.101(noti)[14906] NOTE
> ldmd.c:187:cleanup() Exiting
> 20170828T233527.480585Z ldmd[2785] NOTE ldmd.c:170:reap() child 14906
> exited with status 1
> 20170828T233630.077073Z noaaportIngester[2788] ERROR
> productMaker.c:1223:pmStart() Missing fragment in sequence, last
> 0/178919692 this 2/178919692
> 20170828T233638.371646Z 104.197.198.101(noti)[15249] NOTE
> forn5_svc.c:468:forn_5_svc() Starting Up(6.13.6/5): 20170828233638.317944
> TS_ENDT {{NOTHER, "TIRC"}}
> 20170828T233638.371771Z 104.197.198.101(noti)[15249] NOTE
> forn5_svc.c:471:forn_5_svc() topo:  104.197.198.101 NOTHER
> 20170828T233642.806385Z 127.0.0.1(noti)[15269] NOTE
> forn5_svc.c:468:forn_5_svc() Starting Up(6.13.6/5): 20170828233642.803132
> TS_ENDT {{NOTHER, "TIRC"}}
> 20170828T233642.806434Z 127.0.0.1(noti)[15269] NOTE
> forn5_svc.c:471:forn_5_svc() topo:  127.0.0.1 NOTHER
> 20170828T233739.795093Z 127.0.0.1(noti)[15269] ERROR
> forn5_svc.c:273:noti5_sqf() TIRC02 KNES 282337 PAQ: RPC: Unable to receive
> 20170828T233739.795200Z 127.0.0.1(noti)[15269] ERROR
> forn5_svc.c:554:forn_5_svc() pq_sequence failed: Input/output error (errno
> = 5)
> 20170828T233739.795231Z 127.0.0.1(noti)[15269] NOTE ldmd.c:187:cleanup()
> Exiting
> 20170828T233739.796690Z ldmd[2785] NOTE ldmd.c:170:reap() child 15269
> exited with status 1
> 20170828T233744.578033Z 104.197.198.101(noti)[15249] ERROR
> forn5_svc.c:273:noti5_sqf() TIRC02 KNES 282337 PAS: RPC: Unable to receive
> 20170828T233744.578090Z 104.197.198.101(noti)[15249] ERROR
> forn5_svc.c:554:forn_5_svc() pq_sequence failed: Input/output error (errno
> = 5)
> 20170828T233744.578118Z 104.197.198.101(noti)[15249] NOTE
> ldmd.c:187:cleanup() Exiting

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: MKY-400850
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.