[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20020702: Latency with Cornell feed

Unidata Support wrote:
> 
> ------- Forwarded Message
> 
> >To: <address@hidden>
> >cc: Atmos Support <address@hidden>,
> >cc: <address@hidden>
> >From: "Kevin R. Tyle" <address@hidden>
> >Subject: Latency with Cornell feed
> >Organization: SUNY Albany
> >Keywords: 200207021646.g62GkJu13330 IDD latency
> 
> Hi,
> 
> Several days in the last couple of weeks we've noticed significant
> latencies on the DDPLUS|IDS and HDS feeds from Cornell.  The latencies
> tend to manifest themselves during late morning and afternoon
> hours during weekdays.  Just now, we were seeing latencies of close
> to an hour on these feeds.  Based on the Unidata routing page, the
> problem seems specific to Cornell.  Any ideas?
> 
> Thanks,
> 
> Kevin
> 
> ______________________________________________________________________
> Kevin Tyle, Systems Administrator               **********************
> Dept. of Earth & Atmospheric Sciences           address@hidden
> University at Albany, ES-235                    518-442-4571 (voice)
> 1400 Washington Avenue                          518-442-5825 (fax)
> Albany, NY 12222                                **********************
> ______________________________________________________________________
> 
> ------- End of Forwarded Message

Hi Kevin,

I see from thelma's logs that you are having some problems.  I see some
RECLASS messages that must be from your receiving products that are too
old.  However, for some time there have been log entries about dropped
connections to snow.  For example, in the past 18 hours there have been
25 dropped connections to snow:

/local/ldm% grep snow ldmd.log | grep Connection
Jul 01 23:01:58 thelma.ucar.edu snow[2244]: Connection from
snow.nrcc.cornell.edu
Jul 02 01:12:57 thelma.ucar.edu snow[2477]: Connection from
snow.nrcc.cornell.edu
Jul 02 01:24:15 thelma.ucar.edu snow[2497]: Connection from
snow.nrcc.cornell.edu
Jul 02 02:37:52 thelma.ucar.edu snow[2653]: Connection from
snow.nrcc.cornell.edu
Jul 02 03:14:21 thelma.ucar.edu snow[2734]: Connection from
snow.nrcc.cornell.edu
Jul 02 03:54:15 thelma.ucar.edu snow[2825]: Connection from
snow.nrcc.cornell.edu
Jul 02 04:35:52 thelma.ucar.edu snow[2907]: Connection from
snow.nrcc.cornell.edu
Jul 02 05:23:49 thelma.ucar.edu snow[2983]: Connection from
snow.nrcc.cornell.edu
Jul 02 05:55:23 thelma.ucar.edu snow[3053]: Connection from
snow.nrcc.cornell.edu
Jul 02 07:24:21 thelma.ucar.edu snow[3225]: Connection from
snow.nrcc.cornell.edu
Jul 02 09:15:22 thelma.ucar.edu snow[3440]: Connection from
snow.nrcc.cornell.edu
Jul 02 10:32:27 thelma.ucar.edu snow[3600]: Connection from
snow.nrcc.cornell.edu
Jul 02 10:40:20 thelma.ucar.edu snow[3637]: Connection from
snow.nrcc.cornell.edu
Jul 02 11:11:14 thelma.ucar.edu snow[3691]: Connection from
snow.nrcc.cornell.edu
Jul 02 11:48:31 thelma.ucar.edu snow[3760]: Connection from
snow.nrcc.cornell.edu
Jul 02 12:07:49 thelma.ucar.edu snow[3791]: Connection from
snow.nrcc.cornell.edu
Jul 02 12:41:10 thelma.ucar.edu snow[3873]: Connection from
snow.nrcc.cornell.edu
Jul 02 13:27:21 thelma.ucar.edu snow[3937]: Connection from
snow.nrcc.cornell.edu
Jul 02 14:19:18 thelma.ucar.edu snow[4056]: Connection from
snow.nrcc.cornell.edu
Jul 02 14:41:02 thelma.ucar.edu snow[4115]: Connection from
snow.nrcc.cornell.edu
Jul 02 15:19:40 thelma.ucar.edu snow[4166]: Connection from
snow.nrcc.cornell.edu
Jul 02 15:39:58 thelma.ucar.edu snow[4216]: Connection from
snow.nrcc.cornell.edu
Jul 02 16:19:21 thelma.ucar.edu snow[4321]: Connection from
snow.nrcc.cornell.edu
Jul 02 16:38:09 thelma.ucar.edu snow[4372]: Connection from
snow.nrcc.cornell.edu
Jul 02 16:54:13 thelma.ucar.edu snow[4402]: Connection from
snow.nrcc.cornell.edu

Like I said, the dropped connection problem has been going on for some
time - I'm pretty sure I saw that weeks, if not months ago.  So, perhaps
something else has changed in the past few weeks.  Since 23Z on June 17,
I see only a dozen RECLASS messages from snow, three of which are from
today.

However, the disconnections are problematic by themselves. 
Disconnecting and reconnecting is relatively costly to both hosts since
each must reestablish a place in the queue.  It would be good to address
this problem as well.

What to do...   I will run netcheck from here to there for a days.  It
samples the network with a ping and a traceroute at a user specified
interval.   That might give us an indication of where problems might be
occuring.   If you can, please track when the latencies are occurring so
I can try to correlate them with the netcheck output.  

Unfortunately, problems like this can be very hard to solve.  Let's see
what we can do with you feeding from thelma, and if we can't improve the
situation maybe feeding from a different site would be better.  (We have
actually found that sometimes product latencies are smaller when a
product makes two hops to a site instead of one.)

I'll get back to you tomorrow regarding netcheck output.

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************