Re: [ldm-users] Running your LDM queues out of memory really helps!

Solaris /tmp (swap) has been tmpfs by default for about 13 years now I think.  
Maybe I just don't ingest enough (just the inout from a NOAAPort dish and a 
pared down GFS CONDUIT feed) to notice a difference but when I toyed around 
with putting the queue in /tmp I could not tell that big of a difference.

I can imagine it would make a big difference if you CRAFT and full CONDUIT 
feeds.




-----Original Message-----
From: ldm-users-bounces@xxxxxxxxxxxxxxxx on behalf of Gilbert Sebenste
Sent: Wed 2/27/2008 11:45 AM
To: ldm-users@xxxxxxxxxxxxxxxx
Subject: [ldm-users] Running your LDM queues out of memory really helps!
 
Hello all,

I'm just curious as to how many people run their LDM queues from memory?

Using Fedora Core 7 or 8, the default kernel sets aside memory in a device 
known as "/dev/shm". Do a "df -h" and you'll see it. Depending on the size 
of your physical memory, the default kernel setting gives you about 25% of 
the available memory to put files in as an extra "hard drive", if you 
will. Compiling your own kernel, you can put in as little or as much as 
you want. On my system, I have the following:

% df -k

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      707523888  25768472 645235480   4% /
/dev/sda1               101086     18576     77291  20% /boot
tmpfs                  1815312   1197764    617548  66% /dev/shm

To run the LDM queues out of memory, you need to put your ldm.pq and 
pqsurf.pq files into /dev/shm, and remake them every time you reboot, 
since /dev/shm is really just memory. The easiest way to 
do this with LDM 6.6.X or 6.7.X.X is to go into your ldmadmin-pl.conf 
file in your ~/etc directory with the changes I have in mine:

$bin_path = "$ldmhome/bin";
$etc_path = "$ldmhome/etc";
$log_path = "$ldmhome/logs";
$data_path = "$ldmhome/data";
#$pq_path = "$data_path/ldm.pq";
#$surf_path = "$data_path/pqsurf.pq";
$pq_path = "/dev/shm/ldm.pq";
$surf_path = "/dev/shm/pqsurf.pq";

Note the changes to $pq_path and $surf_path.

Now, when I go to /dev/shm, this is what I see:

% cd /dev/shm
% ls -al
total 1197764
drwxrwxrwt  2 root root          80 2008-02-14 10:22 .
drwxr-xr-x 12 root root        4060 2008-02-14 11:31 ..
-rw-rw-r--  1 ldm  users 1222955008 2008-02-14 10:22 ldm.pq
-rw-rw-r--  1 ldm  users    2347008 2008-02-14 10:22 pqsurf.pq

Since the LDM queues run in memory, you can make a 1.2 GB ldm.pq in just a 
few seconds on a Core 2 processor. Much more importantly, if you have a 
lot of feeds and/or are a relay site, this cuts down on your disk 
read/writes by a TON. For instance, I am getting every Level 2 data site 
minus Alaska and NOP (still trying to figure out what ldmd.conf request 
line I need to get the latter test site), and my load average is around 
.5. before this, it was around 3 on "light" days, but when there's a lot 
of stuff happening around the country, it used to go up to 12! Now, it's 
been cut by a factor of 6. That's HUGE, and it saves my hard drive, too.

My overall load average has dropped by 75% or so since I have done this. 
And thanks to Dave Bukowski at the College of DuPage for suggesting this, 
as well as Tom Yoksas from UNIDATA and Mike Dross from Wright-Weather. 
It really works!

Thoughts?

*******************************************************************************
Gilbert Sebenste                                                     ********
(My opinions only!)                                                  ******
Staff Meteorologist, Northern Illinois University                      ****
E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx                                  ***
web: http://weather.admin.niu.edu                                      **
*******************************************************************************
_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/ 

  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: