[ldm-users] Running your LDM queues out of memory really helps!

[Pete Pokrandt]

Jeffrey Lake - Admin wrote:
> Sorry on the first reply Gilbert going directly to you..
>
> Anybody got idea's how to implement this on CentOS ??
>
> I have the available memory (4GB) but it hardly seems to be used
>
> -Jeff
>   

Jeff,

I just checked on several of my Centos/RHEL machines, and the tmpfs 
appears to be enabled by default on RHEL 3/4 and Centos 4/5 (at least 
those are the ones I checked)

For example, on my primary ingest/relay machine idd.aos.wisc.edu, it's a 
dual opteron w/8Gb of RAM, and I didn't do anything special, but doing a 
df shows

[root at idd ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       72G   17G   52G  24% /
/dev/sda1              99M   74M   21M  79% /boot
none                  3.9G     0  3.9G   0% /dev/shm

I may play around with this too. Probably on local machines first so I 
don't impact people downstream if I mess up!

Pete


>
> Hello all,
>
> I'm just curious as to how many people run their LDM queues from memory?
>
> Using Fedora Core 7 or 8, the default kernel sets aside memory in a device
> known as "/dev/shm". Do a "df -h" and you'll see it. Depending on the size
> of your physical memory, the default kernel setting gives you about 25% of
> the available memory to put files in as an extra "hard drive", if you
> will. Compiling your own kernel, you can put in as little or as much as
> you want. On my system, I have the following:
>
> % df -k
>
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/mapper/VolGroup00-LogVol00
>                       707523888  25768472 645235480   4% /
> /dev/sda1               101086     18576     77291  20% /boot
> tmpfs                  1815312   1197764    617548  66% /dev/shm
>
> To run the LDM queues out of memory, you need to put your ldm.pq and
> pqsurf.pq files into /dev/shm, and remake them every time you reboot,
> since /dev/shm is really just memory. The easiest way to
> do this with LDM 6.6.X or 6.7.X.X is to go into your ldmadmin-pl.conf
> file in your ~/etc directory with the changes I have in mine:
>
> $bin_path = "$ldmhome/bin";
> $etc_path = "$ldmhome/etc";
> $log_path = "$ldmhome/logs";
> $data_path = "$ldmhome/data";
> #$pq_path = "$data_path/ldm.pq";
> #$surf_path = "$data_path/pqsurf.pq";
> $pq_path = "/dev/shm/ldm.pq";
> $surf_path = "/dev/shm/pqsurf.pq";
>
> Note the changes to $pq_path and $surf_path.
>
> Now, when I go to /dev/shm, this is what I see:
>
> % cd /dev/shm
> % ls -al
> total 1197764
> drwxrwxrwt  2 root root          80 2008-02-14 10:22 .
> drwxr-xr-x 12 root root        4060 2008-02-14 11:31 ..
> -rw-rw-r--  1 ldm  users 1222955008 2008-02-14 10:22 ldm.pq
> -rw-rw-r--  1 ldm  users    2347008 2008-02-14 10:22 pqsurf.pq
>
> Since the LDM queues run in memory, you can make a 1.2 GB ldm.pq in just a
> few seconds on a Core 2 processor. Much more importantly, if you have a
> lot of feeds and/or are a relay site, this cuts down on your disk
> read/writes by a TON. For instance, I am getting every Level 2 data site
> minus Alaska and NOP (still trying to figure out what ldmd.conf request
> line I need to get the latter test site), and my load average is around
> .5. before this, it was around 3 on "light" days, but when there's a lot
> of stuff happening around the country, it used to go up to 12! Now, it's
> been cut by a factor of 6. That's HUGE, and it saves my hard drive, too.
>
> My overall load average has dropped by 75% or so since I have done this.
> And thanks to Dave Bukowski at the College of DuPage for suggesting this,
> as well as Tom Yoksas from UNIDATA and Mike Dross from Wright-Weather.
> It really works!
>
> Thoughts?
>
> *******************************************************************************
> Gilbert Sebenste 
> ********
> (My opinions only!)                                                  ******
> Staff Meteorologist, Northern Illinois University                      ****
> E-mail: sebenste at weather.admin.niu.edu                                  ***
> web: http://weather.admin.niu.edu                                      **
> *******************************************************************************
> _______________________________________________
> ldm-users mailing list
> ldm-users at unidata.ucar.edu
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/ 
>
> _______________________________________________
> ldm-users mailing list
> ldm-users at unidata.ucar.edu
> For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/ 
>   


-- 
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
^ Systems Programmer               V Madison,         WI     53706    ^
^                                  V       poker at aos.wisc.edu         ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
^ University of Wisconsin-Madison  V (608) 262-0166 (Fax)             ^
+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+