The LDM Product-Queue
The product-queue is a memory-mapped file that temporarily
stores data-products.
Access to the product-queue is via the pq module, which
coordinates access by the reading and writing processes
to ensure that the product-queue isn't corrupted.
The product-queue has the following characteristics:
- The number of data-products
that the queue can contain is fixed when the queue is created.
This is the maximum number of
data-product slots.
- A data-product is
inserted into the product-queue in contiguous space and indexed
by to the time of insertion according to the system clock.
- Reading a data-product
from the queue creates a read-lock on that product.
Read-locks can be shared by reading processes.
- Data-products
read from the product-queue sequentially are returned in time-sorted
order (oldest first) according to the insertion-time. Therefore,
data-products
can be skipped if the system clock is set backwards.
- Writing a data-product
into the queue creates a write-lock on that product.
Write-locks can't be shared.
- If a new data-product
can't be added,
either because no space is available for the
data-product
or because no slot is available, then
unlocked data-products
are deleted from the queue -- begining with the oldest data-product and
continuing to the youngest -- until addition is possible.
- A writer-counter
field in the product-queue keeps track
of the number of times
the product-queue has been opened for writing. Opening the
product-queue for writing increments the counter; closing the
product-queue decrements the counter. If no process has the
product-queue open for writing, then the writer-counter
should be 0. This counter is used to ensure that the product-queue
has been properly closed by all processes that could modify it.
The canonical pathname of the LDM product-queue, relative to the LDM
home-directory, is data/ldm.pq
A paper on implementation-details can be found
here.
