Rodger, Basically, you want the queue to be large enough so that the minimum residence-time is *at least* the value of the "maximum latency" parameter. The residence-time is the amount of time that a product spends in the queue before being deleted to make room for a new product. The "maximum latency" parameter is the amount of time before "now" that an LDM uses to compute the "from" time when initiating a connection to an upstream LDM. It's also a rejection threshold: products that were created earlier than "now" minus "maximum latency" will be rejected and not inserted into the queue. As a consequence, a correctly configured LDM will reject a data-product that duplicates one already in the queue. The capacity of the queue is based on two parameters: the queue "size" and the number of slots. The queue size is the maximum number of bytes that the queue can contain. The number of slots is the maximum number of products that the queue can contain. Your queue has a maximum latency parameter of 22 seconds. This means that you can be offline for that long without loosing any data. That's not very long. It was computed by the ldmadmin(1) utility based on the minimum residence-time of the queue. If you want to increase that safety margin, then you'll need to increase the capacity of the queue from 500 MB and around 9800 slots. Assuming you'd like a minimum residence-time of an hour, you could set the "/server/max-latency" parameter to 3600 seconds (regutil -u 3600 /server/max-latency) and the "/reconciliation-mode" parameter to "increase queue" (regutil -s 'increase queue' /reconciliation-mode) and then execute the command "ldmadmin vetqueuesize". This is dangerous, however, because extrapolation from such a small queue is prone to error. A less error-prone method would be to iterate up to the final capacity by first setting the reconciliation-mode parameter to "increase queue" and then repeatedly doubling the maximum latency parameter and executing the command "ldmadmin vetqueuesize" until the desired maximum latency is reached. Incidentally, the "ldmadmin vetqueuesize" command won't do anything until the queue is full (i.e., until a product has been deleted) -- so don't be surprised if it doesn't do anything for a while after increasing the maximum latency parameter. One thing to note: the LDM system works best if the entire queue can be kept in physical memory. So be careful if and when the queue size approaches the amount of physical memory (we try not to exceed about 80% of physical memory). Are you using the metrics-collecting and plotting capabilities of the LDM system (e.g., crontab(1)-executed "ldmadmin addmetrics" and the "ldmadmin plotmetrics" command). It's a good way to keep tabs on the LDM in general and the product-queue in particular. Contact me if you have any questions. > I've read the documentation on tuning the product queue parameters > with pqmon and read some of the support inquiries. I think that most > of this is predicated on being IDD sites where latency is the primary > concern. We have a dish and ingest the majority of what we need. We are > using noaaportIngester for ingest. The only IDD feeds we have are for > the lightning data and from FSL for Madis products. I've run 'ldmadmin > vetqueuesize' at various times and let it set the latency. Our latency > is obviously not an issue since we are ingesting directly from NOAAPort > and not feeding any downstream sites. > > I ran pqmon at 5 second intervals for the past few hours. Here are > the stats: > > nprods ~9800 > nfree <10 on average with a peak of 18 > nempty single digits or 0 much of the time but did spike at about 5000 one > time > nbytes ~270000000 - 500000000 > maxprods 9802 > maxfree 43 > minempty 0 > maxext ~10000 - 425000000 > age 25 - 200 > > Here is our config: > > ldmadmin config > > hostname: data.awis.com > os: Linux > release: 4.4.0-31-generic > ldmhome: /usr/ldm > LDM version: 6.13.2 > PATH: > /usr/ldm/ldm-6.13.2/bin:.:/usr/bin:/usr/local/bin:/loc/local/bin:/bin:/loc/wxp_linux:/usr/ldm/bin > LDM conf file: /usr/ldm/etc/ldmd.conf > pqact(1) conf file: /usr/ldm/etc/pqact.conf > scour(1) conf file: /usr/ldm/etc/scour.conf > product queue: /dataspare/ldm/queues/ldm.pq > queue size: 500M bytes > queue slots: default > reconciliation mode: decrease maximum latency > pqsurf(1) path: /dataspare/ldm/queues/pqsurf.pq > pqsurf(1) size: 2M > IP address: 0.0.0.0 > port: 388 > PID file: /usr/ldm/ldmd.pid > Lock file: /usr/ldm/.ldmadmin.lck > maximum clients: 256 > maximum latency: 22 > time offset: 22 > log file: /dataspare/ldm/logs/ldmd.log > numlogs: 7 > log_rotate: 1 > netstat: /bin/netstat -A inet -t -n > top: /usr/bin/top -b -n 1 > metrics file: /dataspare/ldm/logs/metrics.txt > metrics files: /dataspare/ldm/logs/metrics.txt* > num_metrics: 4 > check time: 1 > delete info files: 0 > ntpdate(1): /usr/sbin/ntpdate > ntpdate(1) timeout: 5 > time servers: ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu > otc1.psu.edu timeserver.unidata.ucar.edu > time-offset limit: 10 > > (Note: this server is not sending rstats as it is in test mode). > > So, given that fact that we have only 2 low volume IDD feeds and get > everything else directly from our dish and don't feed any other sites, > how do our numbers look? If needed, we can increase our queue as it > resides on a separate 1TB drive. The drive is a Seagate SSHD (solid > state hybrid) that operates at near SSD speeds. > > As always, your insights are appreciated! Regards, Steve Emmerson Ticket Details =================== Ticket ID: XTK-117583 Department: Support LDM Priority: Normal Status: Closed =================== NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.