Robert, > I have a question regarding duplicate products... > > Let me first describe our LDM processing... we have one "gateway" > server running pdinrs (Planetary Data) and LDM (6.10.1) that processes > the NOAAPort stream received via satellite/Novra receiver. This LDM > instnace essentially acts as a relay to other downstream LDM servers. > It builds an LDM queue (~2GB, product age is about 15-45 minutes), > but does not run pqact. Downstream, our primary LDM server (6.12.3) > is responsible for pulling the products and writing them to file, etc. > This also also has a 2GB queue with a product age of 90-120 minutes. > > My understanding is that incoming product signatures are compared with > the signature of products sitting in the queue. That said, we seem to > commonly experience duplicate products written to file -- a good example > being the LAMP output for a particular hour may show up multiple times in > an appended file. Another example might be the temperature grid from the > RTMA shows up 3 times in a particular model run. This is particularly > bad during times of high retransmission. Using a tool like notifyme > on the downstream server, I can see the same product sitting in queue, > only differing by sequence number. When I extract those 3 instances > of the same product with pqcat, strip off the first 31 bytes, I am left > with 3 identical files, proving (I believe) that the products themselves > are identical. So I am wondering if the product signatures are calculated > with the sequence number? If so, how can I avoid excessive duplication? > Or am I looking at this wrong and is my problem the size of the upstream > queue - should it be larger? There are two problems: 1) the maximum latency parameter and the queue size parameters are inconsistent and need to be reconciled; and 2) the same data-product is being injected into the LDM network multiple times. The first problem is this: when a data-product arrives, it is discarded if its latency (arrival-time minus creation-time) exceeds the value of the registry parameter "/server/max latency"; otherwise, an attempt is made to insert the product into the product-queue. The product will be rejected as a duplicate if a data-product with the same signature (i.e., MD5 checksum) already exists in the queue (the checksum is computed using the product's data: no metadata is involved). For this to be a valid test, the minimum residence time of a product in the queue *must* be greater than the maximum latency time; otherwise, a duplicate product could arrive that would, nevertheless, pass both tests. The second problem is likely due to retransmission requests, which appear to be more numerous with the higher NOAAPORT bandwidth. The fact that the sequence numbers are different in the duplicate products indicates that the same product was transmitted twice. The minimum residence time of a product in the queue can be seen with the command "pqmon -S" (see the manual page on this utility for an explanation). The relationship between the queue size parameters and the maximum latency parameter can be seen or reconciled with the command "ldmadmin vetqueuesize". If the registry parameter "/reconciliation-mode" is set to "do nothing", then the command will inform you of any inconsistency. If, however, this parameter is set to "increase queue" or "decrease maximum latency", then the size of the queue will be increased or the maximum latency decreased, respectively. Be advised, however, that, due to uncertainties in the computation, such an automatic adjustment can be wildly off (resulting in an excessively large queue, for example) if the queue is grossly undersized to begin with. I hope this helps. > Best Regards, > Bob Regards, Steve Emmerson Ticket Details =================== Ticket ID: TUN-623089 Department: Support LDM Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.