[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20020906 UCLA latencies



HI James, 

packetshaper..ugh, but if it truly is only monitoring, then it should not
be an issue.

Traceroute is a one time shot, gives an idea about connection, but is not
overly useful in these type of situations. MTR (Matts Trace Route) is a
more valuable tool as it continues over time. I am not exactly sure where
to download this app, but I am sure a search will find it:

OK it is at:

http://www.bitwizard.nl/mtr/

What we have discovered is that the limiting factor for LDM peformamce is
the # of RPC calls. So oddly enough, we have found that the DDPLUS|IDS
feed can be a killer unless split. Larger prods, although voluminous,
require far fewer RPC calls so less overhead. I am going to investigate
your ldmd.conf file and hopefully have some suggestions for you to
implement. With your bandwidth, we should not have latencies anywhere near
an hour. We will tune this!

More soon!

Thanks,

-Jeff
____________________________                  _____________________
Jeff Weber                                    address@hidden
Unidata Support                               PH:303-497-8676 
COMET Case Study Library                      FX:303-497-8690
University Corp for Atmospheric Research      3300 Mitchell Ln
http://www.unidata.ucar.edu/staff/jweber      Boulder,Co 80307-3000
________________________________________      ______________________

On Mon, 9 Sep 2002, James Murakami wrote:

> Hi Jeff.
> 
> Yes, we've been suffering from high latencies much of the summer. For a while 
> I 
> thought it was TYPHOON's aging disks. So, I started up ldm on a separate, 
> newer 
> workstation (PERIDOT). Things worked fine on it though I never could get the 
> logs to be posted. Sometime in August, things returned to "normal" on 
> TYPHOON. 
> However, a few days before Labor Day, latency problems returned (plagued 
> PERIDOT 
> as well).
> 
> I tried all sorts of combinations, even to the point of ingesting only MCIDAS 
> data. Strangely, latencies last week were still horrible with that little 
> data 
> transmission. Shortly before you emailed me, transmissions seemed faster on 
> both 
> workstations. I increased the data load on both (MCIDAS/DDPLUS/FSL on TYPHOON 
> and UNIDATA/FSL on PERIDOT). It was less than hour latencies on TYPHOON but 
> sometimes over an hour on PERIDOT. Latencies were worse during the day than 
> at 
> night (about 7:30- 8AM, local time, onward into early evening). This has been 
> the case each day that I've monitored the activty (using ldmadmin watch).
> 
> I heard from my systems admin that UCLA is using packetshaper but only to 
> monitor the Internet flow into campus (not restricting data flow). Whenever I 
> do 
> traceroute, there seems to be nothing greatly slow. This is what I got this 
> morning (other days have been better even when latencies exceeded an hour):
> 
> 1528Z 9 Sep 2002
> 
> traceroute to aeolus.ucsd.edu (132.239.114.58), 30 hops max, 40 byte packets
>  1  psnet-ms77.atmos.ucla.edu (128.97.77.231)  1 ms  1 ms  1 ms
>  2  164.67.160.1 (164.67.160.1)  2 ms  2 ms  6 ms
>  3  mathsci--mathsci.backbone.ucla.net (169.232.9.1)  2 ms  2 ms  1 ms
>  4  mathsci--core.backbone.ucla.net (169.232.6.61)  2 ms  1 ms  1 ms
>  5  core--border.backbone.ucla.net (169.232.6.11)  2 ms  2 ms  1 ms
>  6  border--calren.backbone.ucla.net (169.232.35.11)  1 ms  2 ms  2 ms
>  7  ISI--UCLA.POS.calren2.net (198.32.248.29)  3 ms  2 ms  2 ms
>  8  USC--ISI.POS.calren2.net (198.32.248.25)  3 ms  3 ms  3 ms
>  9  UCSD--USC.POS.calren2.net (198.32.248.34)  6 ms  6 ms  6 ms
> 10  198.32.248.186 (198.32.248.186)  7 ms  7 ms  6 ms
> 11  sio-rsm--ucsd-gw.ucsd.edu (132.239.255.145)  9 ms  7 ms  7 ms
> 12  aeolus.ucsd.edu (132.239.114.58)  65 ms  57 ms  86 ms
> 
> traceroute to sunny89.atmos.washington.edu (128.95.89.38), 30 hops max, 40 
> byte
> packets
>  1  psnet-ms77.atmos.ucla.edu (128.97.77.231)  1 ms  1 ms  1 ms
>  2  164.67.160.1 (164.67.160.1)  2 ms  2 ms  1 ms
>  3  mathsci--mathsci.backbone.ucla.net (169.232.9.1)  2 ms  1 ms  2 ms
>  4  mathsci--core.backbone.ucla.net (169.232.6.61)  1 ms  1 ms  1 ms
>  5  core--border.backbone.ucla.net (169.232.6.11)  2 ms  2 ms  1 ms
>  6  border--calren.backbone.ucla.net (169.232.35.11)  2 ms  2 ms  1 ms
>  7  ISI--UCLA.POS.calren2.net (198.32.248.29)  2 ms  2 ms  2 ms
>  8  USC--ISI.POS.calren2.net (198.32.248.25)  3 ms  3 ms  3 ms
>  9  Abilene--USC.ATM.calren2.net (198.32.248.86)  3 ms  3 ms  3 ms
> 10  scrm-losa.abilene.ucaid.edu (198.32.8.17)  14 ms  10 ms  10 ms
> 11  sttl-snva.abilene.ucaid.edu (198.32.8.9)  28 ms  28 ms  28 ms
> 12  hnsp1-wes-so-5-0-0-0.pnw-gigapop.net (198.48.91.77)  29 ms  28 ms  29 ms
> 13  uwbr1-GE3-0.cac.washington.edu (198.107.151.51)  28 ms  28 ms  28 ms
> 14  dustbuster-V23.cac.washington.edu (140.142.153.2)  29 ms  29 ms  29 ms
> 15  sunny.atmos.washington.edu (128.95.89.38)  29 ms  29 ms  29 ms
> 
> 
> Wouldn't the traceroute show significant slowing if packetshaper were 
> limiting 
> the rate of flow? I'm also wondering if the "logjam" is occurring somewhere 
> between the UCLA campus and San Diego (when we're on aeolus.ucsd.edu). It's 
> strange that the latency occurs routinely by mid-morning (local time) and 
> pretty 
> much ends in the dead of night.
> 
> Anyway, any suggestions are welcome. I've attached a copy of my ldmd.conf 
> file. 
> You'll see various commented out data type requests. That's my attempts at 
> getting data without huge latencies. Currently, I see latencies of about 
> 30-40 
> minutes on TYPHOON and PERIDOT.
> 
> James 
> 
>  
> > 
> > Hello James, 
> > 
> > We are using some new diagnostics for network and LDM/IDD performance, and
> > we noticed you are still experiencing some large latencies.
> > 
> > We have come up with some ideas that may decrease latencies and improve
> > performance.
> > 
> > Could you please send me your ldmd.conf file so we can take a look at your
> > requests. Your network connectivity seems to be great, so there is no
> > reason to be experiencing these latencies. We have found some alternative
> > methods for requesting data (using aliases in etc/hosts) that has shown to
> > improve performance. I look forward to working on this with you.
> > 
> > Thank you, 
> > 
> > -Jeff
> > ____________________________                  _____________________
> > Jeff Weber                                    address@hidden
> > Unidata Support                               PH:303-497-8676 
> > COMET Case Study Library                      FX:303-497-8690
> > University Corp for Atmospheric Research      3300 Mitchell Ln
> > http://www.unidata.ucar.edu/staff/jweber      Boulder,Co 80307-3000
> > ________________________________________      ______________________
> > 
> 
> --------------------------------------
> James Murakami
> Staff Meteorologist/Student Affairs
> Department of Atmospheric Sciences
> University of California, Los Angeles
> 405 Hilgard Ave.
> Los Angeles, CA  90095-1565
> 
> 
>    e-mail:  address@hidden
> telephone:  310-825-2418
>       Fax:  310-206-5219
> ---------------------------------------
>