[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20050510: [Fwd: Re: [SCOOP_Version2] "Quick" LDM Question]



Gerry,

If you are using the program "pqinsert" to place a file into your queue
that is identical to one already in the queue (same MD5 checksum), then
it will be rejected with a PQ_DUP (-2) log message. This extends across
feedtypes, since the checksum only takes into account the data contents and
not the feedtype associated with a product.

If you really want the same product in the queue multiple times, then
what I have done is append the file name and/or current time at the end of the 
file
to make the file unique (eg add a meta tag like <filename inserted at `date 
-u`>).
That way, the MD5 calculated on the file contents will be unique.

Otherwise, you could use "ldmsend" to send point to point to a host without 
using
a local queue (the downstream would need an "ACCEPT" line in its ldmd.conf) if
you wanted to direct the flow from the upstream, rather than a downstream 
connection.
That would avoid a local product queue trying to service multiple downstream 
feeders.

Steve Chiswell
Unidata User Support




>From: Gerry Creager n5jxs <address@hidden>
>Organization: AATLT, Texas A&M University
>Keywords: 200505101647.j4AGlmP3014869

>Howdy!
>
>We're using LDM to move a lot of ocean models, NAM-awip32/awip12 and MM5 
>data around, as well as some DDPLUS and CRAFT data, for the SCOOP 
>project.  <Tom knows this; historical background>
>
>We're doing both the 2005 hurricane season and some retrospective 
>storms.  We're running into a problem with inserting files with 
>similar/identical checksums into the queue.
>
>Might y'all be able to shed some light on what we *should* be doing?
>
>Thanks,
>Gerry
>
>-------- Original Message --------
>Subject: Re: [SCOOP_Version2] "Quick" LDM Question
>Date: Tue, 10 May 2005 10:17:50 -0500 (CDT)
>From: David Forrest <address@hidden>
>Reply-To: David Forrest <address@hidden>
>To: Gerry Creager N5JXS <address@hidden>
>References: <address@hidden> 
>                    <address@hidden> 
>                       <address@hidden> 
>      <address@hidden> 
>         <address@hidden> 
>          <address@hidden> 
>          <address@hidden> 
>        <address@hidden> 
>        <address@hidden> 
>      <address@hidden> 
><address@hidden> <address@hidden>
>
>On Tue, 10 May 2005, Gerry Creager N5JXS wrote:
>
>> LDM doesx check based on both checksum AND filename.
>
>Hi Gerry, do you mean something like:
>
>with:  x = "n't"      for pqinsert
>          = ""         for actual transport    ??
>
>One of the problems we noticed in testing was that we couldn't re-pqinsert
>a constant test file, even if it had a different filename.  Justin posted
>a test case on this thread earlier, and Justin and I are sending 'EXP
>test.*time' feeds to work around it.  For pqinsert, it seems the
>data/md5sum has to be different, which caused problems for one of the
>theoretical retrospective cases we considered.
>
>Dave
>-- 
>  Dr. David Forrest
>  address@hidden                                    (804)684-7900w
>  address@hidden                             (804)642-0662h
>                                    http://maplepark.com/~drf5n/
>
>-- 
>Gerry Creager -- address@hidden
>Texas Mesonet -- AATLT, Texas A&M University   
>Cell: 979.229.5301 Office: 979.458.4020
>FAX:  979.847.8578 Pager:  979.228.0173
>Office: 903A Eller Bldg, TAMU, College Station, TX 77843
>
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.