[ldm-users] Running your LDM queues out of memory really helps!

[Arthur A. Person]

Gilbert,

Yes... it's a great way to speed things up and save wear-and-tear on a 
disk if your queue is relatively small and you have enough memory.  The 
primary disadvantage is if you reboot you lose everything in the queue, 
which is a problem for relay sites and a potential problem for the local 
system if the decoders are running behind in queue processing.  I'm 
considering an alternative approach which is to use one of the new SSD 
(Solid State Disk) drives which has the advantage of being fast but also 
retains data on a reboot.  I'm not sure how the pricing is yet, but I 
think it won't be long before SSD's are cheaper than adding main memory. 
My only hesitation is the number of write cycles you get out of the SSD's, 
which is important since and LDM queue is write intensive.  I have to look 
into that a bit futher...

                                 Art


On Wed, 27 Feb 2008, Gilbert Sebenste wrote:

> Hello all,
>
> I'm just curious as to how many people run their LDM queues from memory?
>
> Using Fedora Core 7 or 8, the default kernel sets aside memory in a device
> known as "/dev/shm". Do a "df -h" and you'll see it. Depending on the size
> of your physical memory, the default kernel setting gives you about 25% of
> the available memory to put files in as an extra "hard drive", if you
> will. Compiling your own kernel, you can put in as little or as much as
> you want. On my system, I have the following:
>
> % df -k
>
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/mapper/VolGroup00-LogVol00
>                      707523888  25768472 645235480   4% /
> /dev/sda1               101086     18576     77291  20% /boot
> tmpfs                  1815312   1197764    617548  66% /dev/shm
>
> To run the LDM queues out of memory, you need to put your ldm.pq and
> pqsurf.pq files into /dev/shm, and remake them every time you reboot,
> since /dev/shm is really just memory. The easiest way to
> do this with LDM 6.6.X or 6.7.X.X is to go into your ldmadmin-pl.conf
> file in your ~/etc directory with the changes I have in mine:
>
> $bin_path = "$ldmhome/bin";
> $etc_path = "$ldmhome/etc";
> $log_path = "$ldmhome/logs";
> $data_path = "$ldmhome/data";
> #$pq_path = "$data_path/ldm.pq";
> #$surf_path = "$data_path/pqsurf.pq";
> $pq_path = "/dev/shm/ldm.pq";
> $surf_path = "/dev/shm/pqsurf.pq";
>
> Note the changes to $pq_path and $surf_path.
>
> Now, when I go to /dev/shm, this is what I see:
>
> % cd /dev/shm
> % ls -al
> total 1197764
> drwxrwxrwt  2 root root          80 2008-02-14 10:22 .
> drwxr-xr-x 12 root root        4060 2008-02-14 11:31 ..
> -rw-rw-r--  1 ldm  users 1222955008 2008-02-14 10:22 ldm.pq
> -rw-rw-r--  1 ldm  users    2347008 2008-02-14 10:22 pqsurf.pq
>
> Since the LDM queues run in memory, you can make a 1.2 GB ldm.pq in just a
> few seconds on a Core 2 processor. Much more importantly, if you have a
> lot of feeds and/or are a relay site, this cuts down on your disk
> read/writes by a TON. For instance, I am getting every Level 2 data site
> minus Alaska and NOP (still trying to figure out what ldmd.conf request
> line I need to get the latter test site), and my load average is around
> .5. before this, it was around 3 on "light" days, but when there's a lot
> of stuff happening around the country, it used to go up to 12! Now, it's
> been cut by a factor of 6. That's HUGE, and it saves my hard drive, too.
>
> My overall load average has dropped by 75% or so since I have done this.
> And thanks to Dave Bukowski at the College of DuPage for suggesting this,
> as well as Tom Yoksas from UNIDATA and Mike Dross from Wright-Weather.
> It really works!
>
> Thoughts?
>
> *******************************************************************************
> Gilbert Sebenste                                                     ********
> (My opinions only!)                                                  ******
> Staff Meteorologist, Northern Illinois University                      ****
> E-mail: sebenste at weather.admin.niu.edu                                  ***
> web: http://weather.admin.niu.edu                                      **
> *******************************************************************************
> _______________________________________________
> ldm-users mailing list
> ldm-users at unidata.ucar.edu
> For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/
>
>

Arthur A. Person
Research Assistant, System Administrator
Penn State Department of Meteorology
email:  person at meteo.psu.edu, phone:  814-863-1563