My rule of thumb is that if you're an:
*End node* (i.e not feeding another LDM) -- you need only as much as the
largest product you'll see in the stream plus some overhead. We tend to
dump large model files into the queue, sometimes as large as 800MB. You
want to make sure you can process and save that product to disk so the
queue size here would be about 4 times that size or about 3Gb. But since
you're doing NOAAPort, most of the products are small by that comparison. I
tend to keep a queue size large enough to store no more than 5 min of data
just so I can monitor what's coming across. So I use the default (500MB)
which gives me about 3-5 minutes of storage in the queue.
*Relay node* -- you need a large enough queue to store data for an outage.
At my work, typical outages are between 5-10 minutes, mostly from system
reboots after OS patching. Sometimes we have small networking outages that
last 10 to 15 minutes. We then compute the LDM throughput to get a queue
size. So if you take 15 minutes, to be safe, with about 200MB per minute,
you'd get 3Gb. If you're pushing more than NOAAPort data, this will
increase the size of the queue. We have some systems with systems with
large queues, well over 5GB.
I'm not one for keeping an hour in the queue because the number of products
from NOAAPort would make restarts very slow as the downstream LDM would
spend a lot of time sync'ing all the products from the upstream LDM. So
it's good to find the right balance, large enough to recover from an outage
but not too large to tie up all your disk space or make queue sync'ing
difficult.
Dan.
On Fri, Oct 1, 2021 at 3:52 PM Patrick L. Francis <wxprofessor@xxxxxxxxx>
wrote:
>
> so does that means gilbert, you agree we should keep an hour's worth of
> data in the queue? or is time not that important a factor? :)
>
>
> cheers,
>
> --patrick
>
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
--
*Dan Vietor*
*Senior Research Meteorologist*
CIRA, Colorado State Univ
Aviation Weather Center
Kansas City, MO
816.584.7211
