[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[TIGGE #CZX-469246]: Fwd: tdssingest, CAVES on DSG's VM

Doug,

> I've got a couple of questions regarding a test ldm ingest system for
> TIGGE data.
> 
> We're trying to run a ldm ingester (ingest data and write to disk
> only), without any relay needs,
> on a linux virtual machine.  The way ldm is configured, I think the
> queue is remaining completely in memory (500mb
> on a 6 GB memory system), but is there a way to guarantee that the
> queue resides only in memory without swapping
> to disk?  We're seeing numerous missing fields on the output disk, and
> trying to determine if it is a problem with getting products
> out of the queue before they are purged because of local disk issues,
> or because of features regarding the destination disk.
> The queue is located on local disk, and we're trying to get pqact to
> write files out of the queue to
> a gpfs mounted disk system that provides the necessary capacity for
> TIGGE data.
> 
> Warning messages of this type have been written in the logs:
> 
> >>>> "Nov 16 13:04:40 tdssingest pqact[17362] WARN: Processed oldest
> >>>> product in queue: 123.726 s"

The warning from pqact(1) about having processed the oldest product in the 
queue is definitive.  It means that, while pqact(1) was processing the oldest 
product, the second-oldest product was likely deleted in order to make room for 
an incoming product.

You need to either increase the size of the queue (hoping that pqact(1) can 
keep up with the incoming data flow on average), decrease the rate at which 
data arrives (by reducing the REQUEST, for example), or increase the speed of 
pqact(1).

> Here's a sample of the most recent top:
> 
> top - 15:26:31 up 6 days,  1:22,  1 user,  load average: 1.28, 1.20,
> 0.86
> Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.8%us,  1.0%sy,  0.0%ni, 96.9%id,  0.0%wa,  0.1%hi,
> 1.2%si,  0.0%st
> Mem:   6106980k total,  2174528k used,  3932452k free,   213836k buffers
> Swap: 30716272k total,      104k used, 30716168k free,   728128k cached
> 
> Some of the system hardware specs are included below.
> 
> Thanks,
> Doug
> 
> Begin forwarded message:
> 
> > From: Paul Goodman <address@hidden>
> > Date: November 16, 2009 2:45:03 PM MST
> > To: chifan <address@hidden>
> > Cc: Lynda McGinley <address@hidden>, Doug Schuster <address@hidden
> > >
> > Subject: Re: tdssingest, CAVES on DSG's VM
> >
> > It's actually 4 500GB disks in a hardware RAID configuaration, but
> > yes, the
> > speed is still 7200rpm.
> >
> > There are 6GB of memory allocated to this machine, and it peaks at
> > about
> > 4GB of usage.
> >
> > -Paul
> >
> > chifan wrote:
> >> I think I found the quote that Lynda sent to me for reference.  It
> >> is a
> >> Dell Power Edge R710 with E5530 Xeon
> >> 2.4GHz 8M cache.  There is one 500GB 7.2K RPM SATA drive.  I
> >> suspect we
> >> may be hitting the limit of
> >> access speed at 7.2k RPM.
> >>
> >> What's the peak usage of memory?  I wonder if Doug could use the
> >> memory
> >> for pq.  The size of pq now
> >> is about 1GB.
> >>
> >> Paul Goodman wrote:
> >>> Looks like average write speed is 2MB/sec, with spikes up to
> >>> ~5.5MB/sec.
> >>>
> >>> chifan wrote:
> >>>
> >>>> Paul,
> >>>>
> >>>> What kind of local disk does DSG's VM have?  Is it 7200 rpm SATA or
> >>>> faster SAS?
> >>>>
> >>>> -- Chi-Fan
> >>>> address@hidden, Tel:303/497-1833, Fax:303/497-1291, http://dss.ucar.edu
> >>>> Mail: NCAR, P.O. Box 3000, Boulder, CO 80307-3000
> >>>> Overnight: NCAR, ML 288A, 1850 Table Mesa Drive, Boulder, CO 80305
> >>>>
> >>>>
> >>>
> >>>
> >>
> >
> > --
> > Paul Goodman, 303-497-1235
> > address@hidden
> > NCAR/CISL

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: CZX-469246
Department: Support IDD TIGGE
Priority: Normal
Status: Closed