[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #IJN-200350]: 20060708: EAST and WEST datasets not updating on unidata2

Jerry,

Please forgive my delay getting started on this.  I'll go ahead and configure 
the virtual 
interface, create a gvarldm user and home directory, change the config of the 
current LDM
to bind only to the unidata2 address, and a few other incidental tasks.

mike

> Tom, and Mike,
> 
> Unidata Datastream Support wrote:
> > Hi Jerry,
> >
> > re:
> >> The ldm relay of gvar data has been running since Friday ... so far, after 
> >> the permission problem
> >> got fixed Friday morning, it seems to be working well.
> >
> > I agree.
> >
> >> Have you had a chance to evaluate the
> >> impact on unidata2.ssec.wisc.edu?  Should the queue remain the current 
> >> size?
> >
> > Mike Schmidt and I chatted about the impact on unidata2's LDM, which is a 
> > shorter set of realtime
> > data available in the queue.  We discussed two approaches for mitigating 
> > the impact:
> >
> > - increasing the size of the queue from 2 GB to 3 GB
> >
> > - assigning a second, virtual IP address to unidata2 and running a second 
> > LDM that listens
> >   only to this address
> 
> 
> 
> I have an additional name/IP address to put on unidata2.ssec.wisc.edu
> 
> 128.104.110.24  gvarldm.ssec.wisc.edu
> 
> Mike, did you want me to set that up on hme0:1 or were you going to handle it?
> I'm not sure how to set up a second ldm, so maybe Tom can help with that.
> 
> 
> 
> 
> >
> > The first option is quick and easy to implement, but I wanted to review the 
> > processing
> > you wrote for the GOES data to make sure that resends of the same data 
> > would not
> > harm the ADDE serving of EAST/WEST data.
> >
> > Last night, Mike suggested the second approach as being more attractive:
> >
> >   Date: Mon, 24 Jul 2006 18:34:33 -0600)
> >   To: address@hidden
> >   Subject: moving goes data to unidata2
> >
> >   Tom,
> >
> >   After very little thought, I think we should consider moving GOES data
> >   to unidata2 in the fashion we were discussing for motherlode (via a
> >   virtual IP and binding a second LDM to that address).  This would avoid
> >   over-committing memory resources needed for the operational LDM service,
> >   especially on unidata2 where there's currently no expandability and
> >   currently no free memory.
> >
> >   We'd have to request another IP address from SSEC to make it go.
> >
> >   mike
> >
> >> Currently we have two independent systems ingesting both operational GVAR 
> >> streams(gserve1 and
> >> gserve2).   I only have one of the systems relaying the data via the ldm 
> >> to unidata2.   It would be
> >> nice if both systems were set up to relay the data, so if one system goes 
> >> down, or is taken down for
> >> maintenance, the other system will still be supplying the data.  Do you 
> >> think it be possible for
> >> both systems to relay the data without consuming too much bandwidth?
> >
> > I don't think that bandwidth is much of an issue for the Data Center.  That 
> > being said, LDM 6
> > allows one to setup redundant feed requests and only one, the fastest one, 
> > will result in files
> > actually being transferred.  This is the approach that I would recommend 
> > _after_ bringing up
> > a Virtual interface and running a second LDM.
> >
> >> Also, I can think of a couple of problems with doing this ... while the 
> >> data between the systems is
> >> nearly identical .... a few bits of gvar data may be different on occasion 
> >> between the two systems
> >> .. not very many, but both ingestors are completely independent, and 
> >> unless they are started at
> >> precisely the same moment, there sometimes are a bit or two different on a 
> >> couple images a day.
> >> This would mean that the checksums for the same image will be different on 
> >> the two servers.  The ldm
> >> would treat these as different, and pull both over to unidata2 ... correct?
> >
> > Correct.  The duplicate product detection and rejection only works when the 
> > MD5 checksums are identical.
> >
> >> And this would use
> >> additional bandwidth, and allow for less products to remain in the queue?
> >
> > Yes and yes.
> >
> >> Any ideas, or should we just use one system for now?
> >
> > We would like to request a second IP address for unidata2.  We would then 
> > like to bring up
> > a separate LDM listening only to that interface and dedicated for the GOES 
> > data transfers.
> > After accomplishing that, we will be in a position to think about a 
> > different approach to
> > requesting the data redundantly.  One idea that comes to mind immediately 
> > (unencumbered
> > by the thought process ;-) is calculating the MD5 signature in a different 
> > manner, like
> > from meta data that is available.  By not assigning the signature the value 
> > of the data
> > product checksum, we could insure that the signatures from different 
> > machines would be
> > the same.
> 
> The INDX files are almost always Identical ... so it may be a good candidate 
> for calculating the
> checksum.    Or creating a pseudo checksum from other meta data would be 
> reasonable too.  Maybe the
> INDX and or file name itself could be used, since these are unique for every 
> Image.
> 
> > This would allow the LDM to reject one of the redundantly requested 
> > products.
> > I like this approach (for now) because we do not know apriori which of the 
> > images that
> > are different is "better".
> >
> >> Thanks for any ideas.
> >
> > Please let us know what you think the issues above (i.e., second, virtual 
> > interface;
> > second LDM; different manner of calculating MD5 signature).
> >
> > Cheers,
> >
> > Tom
> 
> Let me know what else I need to do on this end.
> 
> Jerry
> 
> 
> > ****************************************************************************
> > Unidata User Support                                    UCAR Unidata Program
> > (303) 497-8642                                                 P.O. Box 3000
> > address@hidden                                   Boulder, CO 80307
> > ----------------------------------------------------------------------------
> > Unidata HomePage                       http://www.unidata.ucar.edu
> > ****************************************************************************
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: IJN-200350
> > Department: Support Datastream
> > Priority: Low
> > Status: Closed
> >
> 
> 
> --
> Jerrold Robaidek                       Email:  address@hidden
> SSEC Data Center                       Phone: (608) 262-6025
> University of Wisconsin                Fax: (608) 263-6738
> Madison, Wisconsin
> 
> 


Ticket Details
===================
Ticket ID: IJN-200350
Department: Support Datastream
Priority: Normal
Status: Closed