[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20020805: RPC Timed out error ldmping ldm problem

Hi Mike,

There may be more than one thing going on here, I'm not sure.  Let me
break this down a bit:

sunny89: can connect, but after a delay.  Was there a change in the DNS?

And, sunny89 is sending you old products that you are rejecting.  How
often is this happening?  If you notice when this is happening, do a
notifyme to sunny89 and see how timely it is running.  If it's not
behind itself, then something is slowing down the connection, perhaps
packet examination from the firewall.

aeolus:  can't connect, can't contact portmapper.  Is outbound port 388
blocked?  That would be an odd,configuration, but it's possible.  Please
try ping, nslookup and traceroute to aeolus.

suomildm1: no route to host.  Again, please try ping, nslookup, and
traceroute to aeolus.

This may be matter of seeing which protocols work and which don't to
identify what's open and what's not.

Also, it might be interesting to try putting these three hosts in
/etc/hosts on nimbus.  If they're not there already, and this change
causes the connections to occur faster, then something changed about the
DNS.

Please try these things and let me know what happens.

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************




Unidata Support wrote:
> 
> ------- Forwarded Message
> 
> >To: address@hidden
> >From: Mike Leuthold <address@hidden>
> >Subject: RPC Timed out error ldmping ldm problem
> >Organization: UCAR/Unidata
> >Keywords: 200208051509.g75F9nK02703
> 
> Hello,
>         I'm having problems connecting to all my upstream sites.  Some
> sites do eventually connect after a minute or so delay,
> (sunny89.atmos.washington.edu), and some never do (aeolus.ucsd.edu).  This
> problem began within the last week (I was on vacation).  Here is the
> output of ldmping....
> 
> mbus ~]$ ldmping sunny89.atmos.washington.edu
> Aug 05 14:50:59      State    Elapsed Port   Remote_Host           rpc_stat
> Aug 05 14:51:09   H_CLNTED  10.010411  388   sunny89.atmos.washington.edu  
> select: RPC: Timed out
> Aug 05 14:51:44  ADDRESSED  10.001609    0   sunny89.atmos.washington.edu  
> RPC: Timed out
> Aug 05 14:52:09 RESPONDING   0.250059  388   sunny89.atmos.washington.edu
> Aug 05 14:52:34 RESPONDING   0.061898  388   sunny89.atmos.washington.edu
> ....
> 
> [ldm@nimbus ~]$ ldmping aeolus.ucsd.edu
> Aug 05 14:55:59      State    Elapsed Port   Remote_Host           rpc_stat
> Aug 05 14:56:09   H_CLNTED  10.012756  388   aeolus.ucsd.edu  select: RPC: 
> Timed out
> Aug 05 14:56:44  ADDRESSED  10.001547    0   aeolus.ucsd.edu  RPC: Timed out
> Aug 05 14:57:19      NAMED  10.021541    0   aeolus.ucsd.edu  can't contact 
> portmapper: RPC: Timed out
> Aug 05 14:57:55   H_CLNTED  10.018188  388   aeolus.ucsd.edu  select: RPC: 
> Timed out
> Aug 05 14:58:30  ADDRESSED  10.001576    0   aeolus.ucsd.edu  RPC: Timed out
> Aug 05 14:59:05      NAMED  10.021535    0   aeolus.ucsd.edu  can't contact 
> portmapper: RPC: Timed out
> Aug 05 14:59:40   H_CLNTED  10.021512  388   aeolus.ucsd.edu  select: RPC: 
> Timed out
> Aug 05 15:00:15  ADDRESSED  10.007449    0   aeolus.ucsd.edu  RPC: Timed out
> ....
> 
> [ldm@nimbus ~/etc]$ ldmping suomildm1.cosmic.ucar.edu
> Aug 05 15:04:45      State    Elapsed Port   Remote_Host           rpc_stat
> Aug 05 15:04:55   H_CLNTED  10.007760  388   suomildm1.cosmic.ucar.edu  
> select: RPC: Timed out
> Aug 05 15:05:30  ADDRESSED  10.001583    0   suomildm1.cosmic.ucar.edu  RPC: 
> Timed out
>  Aug 05 15:06:00      NAMED   5.031282    0   suomildm1.cosmic.ucar.edu  
> can't contact portmapper: RPC: Unable to receive; errno = No route to host
> Aug 05 15:06:35   H_CLNTED  10.011507  388   suomildm1.cosmic.ucar.edu  
> select: RPC: Timed out
> 
> Any ideas?  Our telecom people had been fooling with a firewall recently,
> but told me that it is currently down.  Sure seems like a firewall issue.
> thanks.
> 
> Mike
> 
> --
> Mike Leuthold
> Atmospheric Sciences/Institute of Atmospheric Physics
> University of Arizona
> address@hidden
> 520-621-2863
> 
> >From address@hidden Mon Aug  5 11:22:25 2002
> >Subject: more on rpc timed out ldm errors
> 
> Hello,
>         Below is some of the ldmd.log file from v5.1.2
> 
> Aug 05 16:00:45 nimbus rpc.ldmd[30531]: Starting Up (built: Aug 31 2000 
> 11:48:33)
> Aug 05 16:00:45 nimbus pqbinstats[30532]: Starting Up (30531)
> Aug 05 16:00:45 nimbus sunny89[30534]: run_requester: Starting Up: 
> sunny89.atmos.washington.edu
> Aug 05 16:00:45 nimbus pqact[30533]: Starting Up
> Aug 05 16:00:45 nimbus cirp[30535]: run_requester: Starting Up: 
> cirp.met.utah.edu
> Aug 05 16:00:45 nimbus sunny89[30534]: run_requester: 20020805150045.970 
> TS_ENDT {{FSL2|UNIDATA,  ".*"}}
> Aug 05 16:00:45 nimbus cirp[30535]: run_requester: 20020805150045.971 TS_ENDT 
> {{NNEXRAD|NMC3,  ".*"}}
> Aug 05 16:00:45 nimbus striker[30536]: run_requester: Starting Up: 
> striker.atmos.albany.edu
> Aug 05 16:00:45 nimbus striker[30536]: run_requester: 20020805155435.266 
> TS_ENDT {{NLDN,  ".*"}}
> Aug 05 16:00:45 nimbus suomildm1[30537]: run_requester: Starting Up: 
> suomildm1.cosmic.ucar.edu
> Aug 05 16:00:45 nimbus suomildm1[30537]: run_requester: 20020805150045.976 
> TS_ENDT {{GPS,  ".*"}}
> Aug 05 16:00:48 nimbus localhost[30635]: Connection from localhost.localdomain
> Aug 05 16:00:48 nimbus localhost[30635]: Connection reset by peer
> Aug 05 16:00:48 nimbus localhost[30635]: Exiting
> Aug 05 16:00:49 nimbus striker[30536]: FEEDME(striker.atmos.albany.edu): OK
> Aug 05 16:00:54 nimbus allegan[30638]: Connection from allegan.nr.usu.edu
> Aug 05 16:01:09 nimbus hailshaft[30655]: Connection from 
> hailshaft.atmo.ttu.edu
> Aug 05 16:01:10 nimbus cyclone[30658]: Connection from 
> cyclone.atmo.arizona.edu
> Aug 05 16:01:10 nimbus cyclone(feed)[30658]: Starting Up: 20020805140109.071 
> TS_ENDT {{NNEXRAD|NMC3|FSL2,  ".*"}}
> Aug 05 16:01:10 nimbus cyclone(feed)[30658]: topo:  cyclone.atmo.arizona.edu 
> NNEXRAD|NMC3|FSL2
> Aug 05 16:01:10 nimbus cyclone[30659]: Connection from 
> cyclone.atmo.arizona.edu
> Aug 05 16:01:10 nimbus cyclone(feed)[30659]: Starting Up: 20020805155435.266 
> TS_ENDT {{NLDN|UNIDATA,  ".*"}}
> Aug 05 16:01:10 nimbus cyclone(feed)[30659]: topo:  cyclone.atmo.arizona.edu 
> NLDN|UNIDATA
> Aug 05 16:01:45 nimbus sunny89[30534]: FEEDME(sunny89.atmos.washington.edu): 
> select: RPC: Timed out
> Aug 05 16:01:45 nimbus cirp[30535]: FEEDME(cirp.met.utah.edu): select: RPC: 
> Timed out
> Aug 05 16:01:45 nimbus suomildm1[30537]: FEEDME(suomildm1.cosmic.ucar.edu): 
> select: RPC: Timed out
> Aug 05 16:03:46 nimbus sunny89[30534]: FEEDME(sunny89.atmos.washington.edu): 
> OK
> Aug 05 16:03:46 nimbus sunny89[30534]: RECLASS: 20020805150346.506 TS_ENDT 
> {{FSL2|UNIDATA,  ".*"}}
> Aug 05 16:03:46 nimbus sunny89[30534]: skipped: 20020805150046.323 (180.183 
> seconds)
> Aug 05 16:04:59 nimbus allegan[31086]: Connection from allegan.nr.usu.edu
> Aug 05 16:05:14 nimbus hailshaft[31087]: Connection from 
> hailshaft.atmo.ttu.edu
> Aug 05 16:05:55 nimbus sunny89[30534]: RECLASS: 20020805150555.030 TS_ENDT 
> {{FSL2|UNIDATA,  ".*"}}
> Aug 05 16:05:55 nimbus sunny89[30534]: skipped: 20020805150454.841 (60.189 
> seconds)
> Aug 05 16:09:04 nimbus allegan[31524]: Connection from allegan.nr.usu.edu
> Aug 05 16:09:19 nimbus hailshaft[31525]: Connection from 
> hailshaft.atmo.ttu.edu
> Aug 05 16:12:49 nimbus striker[30536]: Timed out after 720 seconds inactivity
> Aug 05 16:12:49 nimbus striker[30536]: Disconnect
> Aug 05 16:12:54 nimbus allegan[30638]: Timed out after 720 seconds inactivity
> Aug 05 16:12:54 nimbus allegan[30638]: Exiting
> Aug 05 16:13:00 nimbus rpc.ldmd[30531]: child 30638 exited with status 110
> Aug 05 16:13:09 nimbus hailshaft[30655]: Timed out after 720 seconds 
> inactivity
> Aug 05 16:13:09 nimbus hailshaft[30655]: Exiting
> Aug 05 16:13:09 nimbus rpc.ldmd[30531]: child 30655 exited with status 110
> Aug 05 16:13:09 nimbus allegan[32208]: Connection from allegan.nr.usu.edu
> Aug 05 16:13:20 nimbus striker[30536]: run_requester: 20020805155435.266 
> TS_ENDT {{NLDN,  ".*"}}
> Aug 05 16:13:25 nimbus hailshaft[32209]: Connection from 
> hailshaft.atmo.ttu.edu
> Aug 05 16:13:41 nimbus suomildm1[30537]: run_requester: 20020805151341.339 
> TS_ENDT {{GPS,  ".*"}}
> Aug 05 16:14:01 nimbus cirp[30535]: run_requester: 20020805151401.342 TS_ENDT 
> {{NNEXRAD|NMC3,  ".*"}}
> Aug 05 16:14:20 nimbus striker[30536]: FEEDME(striker.atmos.albany.edu): 
> select: RPC: Timed out
> Aug 05 16:14:41 nimbus suomildm1[30537]: FEEDME(suomildm1.cosmic.ucar.edu): 
> select: RPC: Timed out
> Aug 05 16:15:01 nimbus cirp[30535]: FEEDME(cirp.met.utah.edu): select: RPC: 
> Timed out
> Aug 05 16:16:20 nimbus striker[30536]: FEEDME(striker.atmos.albany.edu): OK
> Aug 05 16:16:20 nimbus striker[30536]: forn_svc_run: select(5, 4, ...): Bad 
> file descriptor
> Aug 05 16:16:20 nimbus striker[30536]: Disconnect
> Aug 05 16:16:50 nimbus striker[30536]: run_requester: 20020805155435.266 
> TS_ENDT {{NLDN,  ".*"}}
> Aug 05 16:16:51 nimbus striker[30536]: FEEDME(striker.atmos.albany.edu): OK
> Aug 05 16:16:59 nimbus allegan[31086]: Timed out after 720 seconds inactivity
> Aug 05 16:16:59 nimbus allegan[31086]: Exiting
> Aug 05 16:17:05 nimbus rpc.ldmd[30531]: child 31086 exited with status 110
> Aug 05 16:17:14 nimbus hailshaft[31087]: Timed out after 720 seconds 
> inactivity
> Aug 05 16:17:14 nimbus hailshaft[31087]: Exiting
> Aug 05 16:17:14 nimbus rpc.ldmd[30531]: child 31087 exited with status 110
> Aug 05 16:17:14 nimbus allegan[32678]: Connection from allegan.nr.usu.edu
> Aug 05 16:17:16 nimbus sunny89[30534]: RECLASS: 20020805151716.986 TS_ENDT 
> {{FSL2|UNIDATA,  ".*"}}
> Aug 05 16:17:16 nimbus sunny89[30534]: skipped: 20020805151615.350 (61.636 
> seconds)
> Aug 05 16:17:30 nimbus hailshaft[32679]: Connection from 
> hailshaft.atmo.ttu.edu
> Aug 05 16:18:31 nimbus sunny89[30534]: RECLASS: 20020805151831.426 TS_ENDT 
> {{FSL2|UNIDATA,  ".*"}}
> 
> --
> Mike Leuthold
> Atmospheric Sciences/Institute of Atmospheric Physics
> University of Arizona
> address@hidden
> 520-621-2863
> 
> >From address@hidden Mon Aug  5 14:22:40 2002
> >To: address@hidden
> 
> Hello,
>         I have been in contact with the networking person here and he
> is out of ideas on how to fix this problem.  He has been sniffing packets
> to see what is going on and sees strange things and would like to discuss
> it with someone at unidata.  It does appear that this problem is
> preventing any downstream sites from getting data from me and is causing
> sporadic data loss here so I would like to fix this soon.  Thanks.
> 
> Mike
> 
> --
> Mike Leuthold
> Atmospheric Sciences/Institute of Atmospheric Physics
> University of Arizona
> address@hidden
> 520-621-2863
> 
> ------- End of Forwarded Message