Re: Pattern in Late Data

• To: Larry Riddle <lriddle@xxxxxxxx>
• Subject: Re: Pattern in Late Data
• From: "Kevin R. Tyle" <ktyle@xxxxxxxxxxxxxxxx>
• Date: Wed, 9 Oct 2002 15:33:40 +0000 (GMT)

Hi Larry,

Like you, we have recently experienced latency and occasional lost
data during these time periods (and also in the hours when the
1200 UTC gridded data is flowing most rapidly).

However, in our case, we believe our problem is local to
UAlbany.  We believe that student use of the internet (most likely
via the use of file-sharing apps) is overwhelming the University's
internet gateway.  The main manifestation of this is sen with large
data packets--such as the HDS feed.  Our ASCII data has not tended
to fall as badly behind; in fact, we haven't seen any evidence of
lost DDPLUS files, such as what you've seen with the METARs.  Despite the
use of traffic-shaping tools such as Packeteer, UAlbany's networking
folks still seem to not have cured the problem (which started
almost to the day that students returned for the fall semester,
and has gotten worse and worse as, presumably, more students
learn the "tools of the trade").  Supposedly LDM traffic is
given priority in the traffic shaper, but it hasn't alleviated
the problem.

How has your maximum latencies looked in your stats files?  Are they
equally bad across all data feeds?  How has your upstream node done?

We find that our data problems tend to disapper after about 0500 UTC
(typically when students go to sleep) and begin to recur around 1500
(when they get up).  Perhaps something similar is happening with
UCSD . . . is there a way for you to find out how much of the
school's bandwidth is being used?

Good luck,

Kevin

______________________________________________________________________
Kevin Tyle, Systems Administrator               **********************
Dept. of Earth & Atmospheric Sciences           ktyle@xxxxxxxxxxxxxxxx
University at Albany, ES-235                    518-442-4571 (voice)
1400 Washington Avenue                          518-442-5825 (fax)
Albany, NY 12222                                **********************
______________________________________________________________________

On Wed, 9 Oct 2002, Larry Riddle wrote:

> For the last few weeks, I've been noticing a pattern in late or missing
> data here at SIO.  Between about 02Z and 05Z my product queue frequently
> gets well behind the power curve.  The METARs (and, probably, every other
> obs type) can be as much as 90 minutes late getting into the hourly files.
>
> Most of the time the system is able to handle this acceptably but, on
> occasion, many of the observations are lost.  This occurred again last
> night.  My 20021009 03Z METAR file, which should be around 550,000 bytes,
> is only 211,206 bytes.  All of today's files from 02Z (possibly as early as
> 01Z) through 04Z are significantly smaller than I would normally expect.
>
> This is probably not a CPU usage problem as the IDD machine
> (aeolus.ucsd.edu) has no other function in life.  All it does is process
> the LDM and build some text files for our weather web page.
>
> Is this problem unique to SIO (and my downstream nodes)?  If so, I'll look
> for a solution locally.  Is anybody else experiencing delays like
> this?  Does this have anything to do with the 00Z HDS product cycle?  If
> this problem isn't unique to the SIO branch of the IDD tree, then I'll buck
> this up to the Unidata support people.
>
> Larry
>
>
>       ---===---=-=-=-=-=-=-=-=-=-=-=====[\/]=====-=-=-=-=-=-=-=-=-=-=---===---
>     -----===(*  Climate's what we expect, but weather's what we
> get.  *)===-----
>    Larry Riddle : Climate Research Division : Scripps Institution of
> Oceanography
>        University of California, San Diego : La Jolla, California  92093-0224
>        Phone: (858) 534-1869 : Fax: (858) 534-8561 : E-Mail: lriddle@xxxxxxxx
>
>
>

• References:
  • Pattern in Late Data
    • From: Larry Riddle