[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20020207: missing products

Harry Edmon wrote:
> 
> Yes.  I don't expect Unidata to fix my disconnect problem (although
> suggestions are certainly welcome).  But can we modify the
> LDM to retry sending the product that timed out instead of skipping
> it.  Perhaps the time sent from the downstream site to the upstream
> site should be backed up a few seconds to make sure nothing is lost.
> After all, it does not hurt to resend a few products - the downstream
> site will eliminate the duplicates.
> 
> 
> --
> Dr. Harry Edmon                 E-MAIL: address@hidden
> 206-543-0547                            address@hidden
> Dept of Atmospheric Sciences    FAX:    206-543-0308
> University of Washington, Box 351640, Seattle, WA 98195-1640

Hi Harry,

Regarding why products are being dropped, here's a scenario that Chiz
was alluding to.  After losing a connection like this, your site is
requesting products from motherlode from the time of the most recent
product in the queue.  This is actually the product's ingest time.  In
general, when feeding from multiple sites, which motherlode does, it's
possible for motherlode to recieve products and order them in its queue
such that the ingest times are not ordered.  Then, if your site,
requests products whose ingest time is more recent than that of some
product that arrived in motherlode's queue *after* the product you are
requesting, you would miss that product. motherlode won't send it
because it's not what you asked for.  This would explain what you're
seeing.

If we modify the software to ask for products a few seconds younger than
the ingest time of the most recent product, we run the risk of sending
duplicates, as you pointed out.  That might be negligable, or it could
be costly if, for example, a bunch of CONDUIT products were being
relayed at that time.  Anyway, it's under consideration.

Having said that, here's a way for you to request products from an
offset different from the time of the most recent product in the queue. 
On the client you could modify the call to rpc.ldmd adding the -m and -o
options.  For example, 'rpc.ldmd -m 3600 -o 5' would send a RECLASS for
products older than one hour, and would also request products from 5
seconds earlier than _now_ no matter what the state of the queue is. 
That's not the same as 5 seconds older than the most recent product in
the queue, but it may still work.  With a steady stream of products an
overlap in the stream would occur and duplicates would be rejected. 
With a pause of greater than 5 seconds it wouldn't matter anyway.  Maybe
it's possible a product with a timestamp of 5 seconds minus some epsilon
would be missed...  

I know it works like this when the ldm is first started, but I don't
know if it would do this in the case of a dropped connection since I've
not been able to test that.  From the man page it seems like it would
work.  Just a thought.  If you try this please let me know how it turns
out.

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                  P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************