Leonard, > I'd like to understand the queues better. I'm getting the idea about > sizing the Upstream queue according the size of individual > data-products, the rate they are inserted and the bandwidth heading > downstream. On the Downstream size, I think it should be sized > according to Upstream and the rate for which they can be disposed of by > pqact and whatever is triggered by that. The product-queue should be sized so that any given data-product will reside in the queue for at least as long as the "maximum latency" parameter, which sets the maximum amount of time from data-product creation to data-product arrival that a downstream LDM will accept for a data-product (a downstream LDM will reject data-products -- and not put them into the product-queue -- if their latency is greater than the maximum latency parameter (which is, typically, 3600 seconds). Beginning with LDM 6.9, the ldmadmin(1) script can be set to automatically reconcile the size of the product-queue and the maximum latency parameter. > I'm timing the insert according to the bandwidth, so that the need for > queuing is minimized. I worry that the network could be down, and the > queue fills, and older data-products are deleted before being sent. > I've seen that happen, indicated in the logs. That's when I decided to > time the inserts instead of inserting everything as quickly as possible > (for one instrument we have, the data are sent overnight, and 100GB is > online to be sent at the start, but I didn't want to create a queue of > 100-200GB). The LDM system is designed for a network that doesn't go down that often -- and not for long if it does. That is why the default maximum latency parameter is one hour. If your network is down for longer periods, then you have a problem. > I'm starting to understand the FEEDME and HEREIS, for example. I see > how Downstream requests data-products from Upstream. But, shouldn't > Downstream notify Upstream that it has received a particular > data-product? How do the two queues work together so as to not send the > same data-product? Each data-product has a unique MD5 checksum. The product-queue will reject any attempt to insert a data-product with an MD5 checksum identical to an existing data-product. > Thanks. > > -- > ==Leonard E. Sitongia > High Altitude Observatory > National Center for Atmospheric Research > P.O. Box 3000 Boulder CO 80307 USA > address@hidden voice: (303)497-2454 fax: (303)497-1589 Regards, Steve Emmerson Ticket Details =================== Ticket ID: EAN-986598 Department: Support LDM Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.