[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20040113: LDM Buffering Question

Chris,

>Date: Tue, 13 Jan 2004 12:04:10 -0600
>From: "Chris D Gilbert" <address@hidden>
>Organization: US DoC/NOAA/NWS/NEXRAD OSF
>To: Steve Emmerson <address@hidden>
>Subject: Re: 20040113: LDM Buffering Question

I thought the Operational Support Facility (OSF) had been renamed the
Radar Operations Center (ROC).

> Thanks for the info. Yes, I had my "upstream" and "downstream"
> concepts reversed.  Thanks for being patient with me,

Ditto.

> I'm new to LDM. Let me give you some background info.  I'm working
> on the "upstream" part of the NEXRD2 feedtype.  Our software will
> be replacing the CRAFT implementation and supply the "water for the
> stream".

So, you're handling the LDM-s that will be at the actual radar sites?

> I found LDM documentation that states: "The LDM is designed so that a
> relay site can be shut down and restarted, then catch up from where it
> left off by getting the products it missed from the upstream host so
> that downstream sites will still receive all the subscribed products."
> 
> This statement worries me when it comes to our remote DOD radar
> sites. We have limited bandwidth on a Frame Relay circuit that
> must be shared.

I believe the radar-to-regional HQ bandwidth is 128 kbps, if I'm not
mistaken.

> If the downstream product-queue tries to catch up
> with an hours worth of data, the Frame Relay circuit can easily
> be overwhelmed.  We will have to run some tests to figure out our
> threshold on data that is "backed up"...

Information on sizing the product-queue can be found at

    
http://my.unidata.ucar.edu/content/software/ldm/ldm-6.0.14/basics/vet-ldmadmin.html

See the description of the "pq_size" variable.

In your case, the maximum rate of data-creation is approximately
(assuming a 6:1 compression-ratio):

    (2432 bytes/radial)(365 radials/scan)(17 scans/4.1 minutes)(1/6)

        ~= 613437 bytes/minute

        ~= 10224 bytes/second 
        
        ~= 81792 kbps

(In practice, compression ratios better than 6:1 are common.)

This rate of data-generation is about 64% of the available bandwith, so,
yes, you'll have to keep the product-queue fairly small in order not to 
overwhelm the connection (crossing your fingers might not be a bad idea,
either :-).

> So, Is resizing the "upstream" product-queue the only way to control
> the amount of data the downstream product-queue can get when a "back
> up" occurs?
>
> I've heard you may be able to set some type of flag on the downstream
> product-queue to not request any old data; but I really need to be
> able to control this from the upstream server.

Because you can't control what the downstream LDM will request,
your best bet is to limit what the upstream will supply.  As the
previously-mentioned URL indicates, you can limit the number of products
that a product-queue can hold.  In your case, you might want to limit
that number to the number of products in a single volume-scan, i.e.,

    (3 products/scan)*(17 scans/volume) = 51 products/volume

This should result in a product-queue whose oldest product was 4.1
minutes, on average.  This can be verified via the pqmon(1) utility.

I hope this helps.

> Thanks again,
> Chris Gilbert

Regards,
Steve Emmerson
LDM Developer