[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20020305: moving the ldm.pq file?

Chris Herbster wrote:
> 
> Just a quick sanity reply on my end ... and a few additional comments.
> 
> First, I know you still think the same of me, for better or worse (-: ....
> 
> Second, the pq file is most certainly on a local disk and always has been.  I 
> *try* to follow directions!  (-:  The disk in question is exported from 
> supercell.
> 
> I was thinking more in terms of hardware efficiency and software overhead 
> when I wanted to move the pq file.  I have moved the pq file over to a 
> different physical disk that is on supercell.  I think this disk is actually 
> on a different controller in the machine too.  My thought is that this might 
> improve the performance of the data disk to the NFS client machines - which 
> are about to increase in number from just a few to many when we get into a 
> new building.
> 
> I already had deleted and recreated the pq file after the reboot of the 
> server.  Since we're now running normally and pretty close to up-to-date with 
> the data, I didn't want to request data we already have.
> 
> I just copied the queue to the other partition (with the LDM down) and have 
> restarted with what appears to be no problems.
> 
> I've also tried something that may not be strictly kosher.  I was reading in 
> the support archives about the issue of number of products per second (RPC 
> calls) being a possible bottle-neck.  What I have tried is splitting our feed 
> between thelma and pluto.met.fsu.edu.  I am requesting the HDS feed from FSU 
> and all the other stuff from thelma.  Before I had no problems with 
> everything but HDS via thelma.  When I added the full HDS feed (I know that 
> was ambitious), then we would do fine at night but fall behind during the 
> day.  I even tried the trick of adding the request using IP number rather 
> than name (and appear to have opened a second connection), but
> lagged behind anyway.
> 
> We had been having trouble keeping up with a full feed from FSU too.  When 
> they failed back in Jan, we started feeding from thelma.  I know that we 
> recently cleared up a bug in our ISP's routers and our throughput has gone up 
> by a lot - as much as 10x.  I was considering trying to go back to FSU for 
> the full feed, but thought I'd try that in steps.
> 
> Let me know if I'm doing something I shouldn't and I'll stop.
> 
> Thanks!
> 
> CH

HI Chris,

This all sounds fine.  Indeed, you're sounding quite like a knowledgable
LDM user!

Moving the queue under the control of a different controller might
improve things.  Would you let me know if you notice a change?

And, we've had some good results splitting feeds so that's a very
reasonable thing to do.  This is because if set up properly it does
indeed create additional connections allowing greater throughput.  If
you know enough about a feed you can write regular expressions to allow
you to split it multiple ways and thus have multiple connections.  There
is a cost to this, however, that is not just local to your machine. 
I.e., it will spawn multiple rpc.ldmds on both the local machine the
upstream host(s).  This is why we've not publicized this more widely. 
But, it sounds like the splitting you've done is very minimal.

I'd also like to know how things go when you switch back to FSU.  

Aren't you getting a dish sometime in the future?

All in all, it sounds like you're doing great!  LDM on!

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************