[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Support #JMD-711173]: Re: LDM question

Dave,

re:
> One question I have for the LDM experts is whether there is a limit to
> the number of REQUEST streams.

No and yes.  There is no programmed limit to the number of requests you
can have.  There is a practical limit on what you should do since each
process receiving products into the LDM queue will use a write lock
for each product received.  Write locks are "expensive" (by comparison,
read locks are "cheap"); if you have too many processes attempting to
perform a write lock on the queue at the same time, your system will
get bound up.  Our experience tells us that the best approach is to
just just as many requests as is needed to get the data with minimum
latencies and no more.  What can be done by 5 requests is _not_
better done with 10, etc.

> Could I specify each file type explicitly? Otherwise is there some way
> to ensure that streams I set up are
> more evenly divided
> in the amount of work each has to do?

As I just wrote to Baudouin, Manuel, etc., our recommended approach
is to include a monotonically increasing sequence number in the
header of each product, and then split the feed based on the sequence
number.  This is exactly what was done during the testing phase, and
we were able to move up to 17 GB/hr from ensembe to dataportal during
those tests with minimal latency.

> Originally I had the pressure
> level variables set up so that each
> stream handled one pressure level for all pressure-level variables.
> But the best set-up seems to be
> dependent on the order in which products are inserted into the upstream
> queue.

Exactly!  This is what I mean about including a monotonically increasing
sequence number in the header.

> Perhaps I just need to
> know more about the insertion order of the products.

The insertion order should not be important at all.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: JMD-711173
Department: Support IDD TIGGE
Priority: Normal
Status: Closed