[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20030715: IDS|DDPLUS latencies and building an LDM with large file support

>From: Celia Chen <address@hidden>
>Organization: NCAR/RAP
>Keywords: 200307151639.h6FGdSLd007355 LDM IDD rtstats largefile

Celia,

>We notice that there seem to be some delay in the METAR data feed
>on RAP's main LDM server chisel. Could you check to see if this 
>is caused by our upstream thelma?

We would be able to better assess/help troubleshoot this situation if
chisel was reporting real time statistics.  Assuming that you are
running a current release of the LDM, please add the following line to
your ~ldm/etc/ldmd.conf file and then stop and restart your LDM:

exec    "rtstats -h rtstats.unidata.ucar.edu"

This should be the last 'exec' line in your ldmd.conf file.

After making the change above, please send along your ldmd.conf file so
we can take a look at how you have your requests setup.

The IDS|DDPLUS feed latencies (IDS|DDPLUS carries METAR data) on thelma
is mostly sub second.  In addition, a machine that thelma is feeding at
San Jose State University, rossby.met.sjsu.edu, is showing IDS|DDPLUS
latencies that are virtually identical to those seen on thelma.

You can see the latency information I referring to in:

thelma IDS|DDPLUS
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+thelma.ucar.edu

SJSU IDS|DDPLUS
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+rossby.met.sjsu.edu

Given the latencies we are seeing on thelma and rossby, I suspect that
any delay you are seeing is a result of how you have your ldmd.conf
request lines setup.  This is why we need your real time statistics
and need to see your ldmd.conf file entries.

>I have recently increased the pq_size to 2GB on chisel in order to get 
>the CONDUIT feed. I tried to increase the pqsize to something greater
>than 2GB to see if it will help the METAE data feed issue, but without 
>success.  Here are the error messages: 
>
>chisel:/home/ldm:51%ldmadmin mkqueue
>Jul 15 16:26:04 pqcreate[3971]: assertion "pq->ixo > pq->datao" failed: file "
> pq.c", line 3736
>Jul 15 16:26:04 UTC chisel.rap.ucar.edu : make_pq: mkqueue failed
>  
>Any help will be greatly appreciated.

You don't say what kind of machine chisel is (Jeff Weber may know, but
I don't).  In order to be able to create an LDM queue larger than 2 GB,
you must build the LDM with large file support, and this not possible
on all operating systems.  The web page that has information on building
an LDM with large file support can be found in:

Unidata HomePage
http://my.unidata.ucar.edu/content/software/ldm/index.html

  LDM 6.0.13
  http://my.unidata.ucar.edu/content/software/ldm/ldm-6.0.13/index.html

    LDM Installation Steps
    
http://my.unidata.ucar.edu/content/software/ldm/ldm-6.0.13/install/install.html

     Source-Code Installation
     instructions for building an LDM that supports large product-queues
     
http://my.unidata.ucar.edu/content/software/ldm/ldm-6.0.13/install/huge-queues.html
    
Tom Yoksas