Mike, pqact(1) uses the signature (i.e., MD5 checksum) of the last, successfully-processed data-product from a "*.state" file to determine where to start processing data-products from the product-queue. If the corresponding product exists, then pqact(1) will start just after that product; otherwise, pqact(1) will start with the product whose age in the product-queue in seconds is given by the argument of the "-o" option. The default is zero, i.e., start with the most recent product. On another issue, specifying a time-offset of 24 hours for the LDM system will only be effective if the product-queue of the upstream LDM holds at least 24 hours worth of data. > Hello Steve, > > I was wondering if you could help me clear up one more point of > understanding with LDM. > > The way UNAVCO treats data is that it's timeless and needs to be collected > regardless of delays. I know this is different than the LDM model, > but I think I have the tuning set to work for us. After an outage I was > using the fields time-offset and max-latency in the registry.xml file to > have the downstream LDM request old queued files for up to 24 hours past > current time. This however was not producing the results I expected. > I found the *.info files in the ldm home directory and tried to clear > those so that there were no indications of the last file processed by > the downstream LDM. > > This still didn't help. Next I cleared the downstream product queue and > still had unexpected results. What I finally discovered is that in the > ldmd.conf file where pqact gets forked, needed to have add the -o flag > set with a value to match the max-latency value. Now that I have all the > settings I need I believe I understand how the LDM process was working. > The downstream was requesting the data from upstream correctly but then > pqact was considering the data too old because the default offset for > pqact was only around 10 minutes. When pqact saw old data it sent a > 'reclass' message I believe to tune the data request with another > feedme request to the upstream LDM starting at the current time minus > ten minutes. This is what I gathered from the entries in the log file. > > I think I now have everything appropriately tuned so that after an > outage our downstream LDM should go back in time up to 24 hours to > retrieve products starting with the last file processed as logged in the > .info file. Performance is good under this configuration so I'm happy > with the version of LDM I am running. What I was most curious about is > could you confirm my understanding of the behavior of LDM and can you > give me a reason why it's advantageous for pqact to run on it's own flags > in the ldmd.conf file and not take it's offset from the registry.xml file. > > Thanks > > -Mike Regards, Steve Emmerson Ticket Details =================== Ticket ID: AZV-974943 Department: Support LDM Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.