[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #UEF-429372]: LDM - pqinsert error

> Institution: National Weather Service
> Package Version: 6.4.4
> Operating System: RHEL WS Ver. 4u2
> Hardware Information: Dual Xeon 3.0GHZ with 3GB DDR
> Inquiry: I am attempting to ldm a series of images to multiple sites (GIF).  
> My image sizes vary 1KB to 35KB and I receive the following error from some 
> of the smaller gif images (Seems to be an MD5 Sum Error)
> pqinsert ERROR: Product already in queue: 8a9f53fd4c568dcda50a353adf929f84    
>  1798 20060202191503.218     EXP 000  
> /data2/www/html/ridge/Warnings/Long/FWS/FWS_20060202_1805_Long_Warnings.gif
> I get this error even though the files are physically different.

The files might be physically different, but the data they contain are 
identical.  This is why the product-queue is rejecting the insertion.  
pqinsert(1) computes an MD5 checksum for the data-portion of every 
data-product.  If two data-products have the same MD5 checksum, then the odds 
of them having different data is about 1:2^128, against.

If the data-products are distinct, then they should have different MD5 
checksums.  For example, you could add a timestamp to the data to indicate when 
it was observed.  A simple counter or filename would also suffice -- anything 
that would make the data different.

Having different product-identifiers (i.e., filename) isn't enough because 
product-identifiers are used by the LDM system but not, in general, by decoders 
of the data.  They must be able to recognize data-products from the data itself 
and not rely on the product-identifier.

> Is there a way to force pqinsert to create a new value so that the image can 
> be sent through the ldm stream.  If I purge the queue and send this file 
> first it works, but when I send the next file it says it is already in the 
> queueu.

You could run pqexpire(1) on the queue, but that really won't help because the 
the downstream LDM will also reject data-products that have identical MD5 

> Thank you for your time,
> Paul Kirkwood
Steve Emmerson

Ticket Details
Ticket ID: UEF-429372
Department: Support LDM
Priority: Normal
Status: Closed

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.