LDM 6.5.1 Release Notes

• Changes from LDM 6.5.0

• Changes from LDM 6.4.6

• Improved LDM performance:
  • Of downstream process during startup by changing product-queue access-mode from writing to readonly when searching backwards for matching product to avoid file write-locking contention.
  • Of upstream process during termination by
    • Adding potential termination point to each data-product during initial backward search of the product-queue.
    • Closing downstream connection upon receipt of SIGTERM.
  • Of downstream process during termination by closing downstream connection upon receipt of SIGTERM.
• Modified the "pqact" utility:
  • Corrected behavior of the "-overwrite" option of the FILE and STDIOFILE actions.
  • Ensured proper behavior of the "-log" option of the FILE and STDIOFILE actions regardless of operating-system.

• Modified top-level LDM server:
  • Replaced sending of SIGINT to process-group with sending of SIGTERM because that allows for a cleaner shutdown of pqact(1) processes. This *shouldn't* affect decoders because they've always been required to terminate upon reception of either a SIGTERM or SIGINT. You might check your decoders, however.
  • Added the ability for the configuration-file to have "include" statements.
• Made all LDM programs safer in the face of asynchronous termination signals (SIGTERM) by eliminating calls to unsafe functions by signal handlers and correcting responses to SIGTERM. The LDM server and its child processes will now respond synchronously to a SIGTERM rather than in an unsafe, asynchronous manner. This might make termination of a running LDM system take longer, but it greatly reduces the risk of mysterious crashes.
• Modified pqact(1) utility:
  • Added saving of the insertion-time of the last successfully-processed data-product in a file whose pathname is that of the configuration-file with ".state" appended. This allows the pqact(1) process in a subsequent LDM session that uses the same configuration-file to start processing where the previous one left-off (it logs this fact). This means that one pqact(1) process, at most, should execute any pqact(1) configuration-file.
  • Added "-log" option to FILE and STDIOFILE actions. This option will cause the pathname and product- identifier to be logged at the NOTICE level.
  • Added logging of warning message when the oldest product in the product-queue is processed.
  • Modified error-message logged with a child process terminates. If the child process was stated via an EXEC entry in the LDM configuration-file, then the command-line is appended to the error-message to identify the entry.
  • Added printing of the command-line of decoders that do not read from standard input quickly enough to log message. This should ease the identification of slow decoders.
  • Increased the number of arguments in a PIPE command from 15 to _POSIX_ARG_MAX/2 (2048).
  • Modified algorithm for determining the canonical time of a WMO data-product from the day-of-month component in the product-identifier. It now assigns the data-product to the previous month if and only if the canonical time of the data-product is one or more days in the future (as determined by the creation-time of the data-product and the day-of-month component in the product-identifier). This should eliminate the problem of old, retransmitted WMO data-products being assigned to a future time.
• Added "-i" option to pqinsert(1). This option causes the MD5 signature to be computed from the product-identifier rather than from the product's data. This will greatly increase the rate at which large data-products can be inserted but requires that the product-identifier be unique.
• Modified behavior of product-queue module:
  • Modified pq_setCursorFromSignature(3) in file "pq/pq.c". It used to find the data-product with the given signature by searching forward in the time-map from just before the creation-time of the target data-product until it either found a time-map entry whose signature matched or until the data-products appeared to come from the same source but were created one minute later. Now it searches forward in the time-map from the same starting point until it finds an entry whose offset to the data region equals that of the signature-map entry. If no entry is found, then a second search is made starting from the beginning of the product-queue up to the starting point of the first search. This second search is much more time-consuming but will find data-products whose creation-time is, otherwise, too far in the future (relative to the local system clock). This should fix Art Person's problem of not finding the "signature" data-product during a reconnection by a downstream LDM (JIG-686458).
  • Added warning-level log message to pq_insert(3) about data-products that are created sufficiently far in the future (see the previous paragraph) to cause problems during reconnection. Only one such message will be logged per hour per ingest host. In this day and age, any computer that's connected to the Internet can easily have an accurate system clock -- and that's been an explicit requirement for proper operation of the LDM from the beginning.
• Modified "log_log(3)" and "err_log(3)" so that they print error- messages using a "%s" format instead of passing them directly to ulog(3) because the messages might contain formatting characters (e.g., "%" from, for example, a product-identifier). This will prevent upstream and downsteam LDM-s from crashing when printing a product-identifier that contains a "%". (Thanks Harry.)
• Fixed bug in upstream LDM that could cause it to skip over data-products. The bug was due to the way an upstream LDM processed a request with an encoded signature. The LDM would adjust the "from" time of the request to the insertion time of the associated product, if found. Under sufficient latency conditions, the insertion time would be after the creation- time of the next products that should be sent and the products would be filtered-out. The correction involves removing the adjustment but still setting the product-queue cursor based on the signature product.
• Improved downstream LDM. Removed from a reconnection attempt the setting of the metadata of the last successfully-received data-product from a search of the product-queue if the number of downstream LDM was two or more. This was mistakenly introduced in version 6.4.5.