The LDM Product-Queue

The product-queue is a memory-mapped file that temporarily stores data-products. Access to the product-queue is via the pq module, which coordinates access by the reading and writing processes to ensure that the product-queue isn't corrupted.

The product-queue has the following characteristics:

• The number of data-products that the queue can contain is fixed when the queue is created. This is the number of data-product slots.
• The amount of data that the queue can contain is fixed when the queue is created. This is the size of the queue.
• A data-product is inserted into the product-queue in contiguous space and indexed by the time of insertion according to the system clock.
• A data-product will not be inserted into the product-queue if its signature (i.e., MD5 checksum) equals that of a product already in the queue.
• Reading a data-product from the queue creates a read-lock on that product. Read-locks can be shared by reading processes.
• Data-products read from the product-queue sequentially are returned in time-sorted order (oldest first) according to the insertion-time. Therefore, data-products can be skipped if the system clock is set backwards.
• Writing a data-product into the queue creates a write-lock on that product. Write-locks can't be shared.
• If a new data-product can't be added, either because no space is available for the data-product or because no slot is available, then unlocked data-products are deleted from the queue -- begining with the oldest data-product and continuing to the youngest -- until addition is possible.
• A writer-counter field in the product-queue keeps track of the number of times the product-queue has been opened for writing. Opening the product-queue for writing increments the counter; closing the product-queue decrements the counter. If no process has the product-queue open for writing, then the writer-counter should be 0. This counter is used to ensure that the product-queue has been properly closed by all processes that could modify it.

The canonical pathname of the LDM product-queue, relative to the LDM home-directory, is var/data/ldm.pq

A paper on the implementation-details is

Rew, Russ and Anne Wilson, 2001: The Unidata LDM System: Recent Improvements for Scalability, Proceedings, 17th IIPS, American Meteorological Society, January, Albuquerque, New Mexico.