[ldm-users] Running your LDM queues out of memory really helps!

[Arthur A. Person]

Gilbert,

On Wed, 27 Feb 2008, Gilbert Sebenste wrote:

> On Wed, 27 Feb 2008, Arthur A. Person wrote:
>
>> Gilbert,
>>
>> Yes... it's a great way to speed things up and save wear-and-tear on a disk
>> if your queue is relatively small and you have enough memory.  The primary
>> disadvantage is if you reboot you lose everything in the queue, which is a
>> problem for relay sites and a potential problem for the local system if the
>> decoders are running behind in queue processing.
>
> Since I have multiple machines, I avoid that problem by rebooting one
> machine at a time, and try to do it in low-volume hours.
>
> The solid state drives, I have been told, have a nasty habit of going out
> on you gradually after about 300,000 writes (which ain't much when you
> have something like NOAAport).

Actually, for 50 GB/day in a 2 GB queue, that would be 25 re-writes/day or 
300000/25/365=32 years.  Some of the newer drives have write cycles in the 
1-5 million range, so I think it might work okay for an LDM queue.

> Also, right now they're about 15
> times more expensive than standard hard drives, have much lower capacity
> than hard drives,

True, but we should be comparing them to system memory, not hard drives. 
8 GB of system memory for one of my servers is selling for ~$600 right 
now.  I can buy a RiDATA 32 GB SATA SSD for ~$680, which is ~ 1/4 the 
price/GB for my server.  Depending on your server and its memory costs, 
SSD could be more expensive, but SSD's are advancing rapidly and I expect 
costs to drop quickly.

> and as I see on Wikipedia, really bad things can happen
> if your power goes out while it's running, even worse than with a hard
> drive. See:
>
> http://en.wikipedia.org/wiki/Solid-state_drive

Hmmm, not sure about the power issue.  If it's a significant problem, I 
would expect the designers would fix it pretty quickly since these things 
are going to be in everyone's laptops pretty soon. Short of breaking (such 
that it's not revivable after a power failure/reboot cycle), I don't see 
how that would be any worse than using main memory... you would still have 
to remake the LDM queue.  Extra main memory is also used extensively by 
the OS for file system caching, so before grabbing up a lot of it for use 
by an LDM queue, you need to be sure you have enough to spare.


                              Art


> *******************************************************************************
> Gilbert Sebenste                                                     ********
> (My opinions only!)                                                  ******
> Staff Meteorologist, Northern Illinois University                      ****
> E-mail: sebenste at weather.admin.niu.edu                                  ***
> web: http://weather.admin.niu.edu                                      **
> *******************************************************************************
> _______________________________________________
> ldm-users mailing list
> ldm-users at unidata.ucar.edu
> For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/
>
>

Arthur A. Person
Research Assistant, System Administrator
Penn State Department of Meteorology
email:  person at meteo.psu.edu, phone:  814-863-1563