Re: [ldm-users] XFS small file performance and prevent Linux OOM from killing LDM

  • To: Gilbert Sebenste <gilbert@xxxxxxx>
  • Subject: Re: [ldm-users] XFS small file performance and prevent Linux OOM from killing LDM
  • From: daryl herzmann <akrherz@xxxxxxxxxxx>
  • Date: Mon, 10 Oct 2016 14:21:37 -0500
  • Authentication-results: spf=none (sender IP is ) smtp.mailfrom=akrherz@xxxxxxxxxxx;
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99

Eh, let us not go down that path! There have been a number of which-filesystem-is-best threads on ldm-users in the past.

daryl

On Mon, 10 Oct 2016, Gilbert Sebenste wrote:

Ah, OK, thanks. Once this is done, how does XFS compare to the reliability and performance of EXT3/4, in your opinion?

Gilbert

-----Original Message-----
From: daryl herzmann [mailto:akrherz@xxxxxxxxxxx]
Sent: Monday, October 10, 2016 2:17 PM
To: Gilbert Sebenste <gilbert@xxxxxxx>
Cc: ldm-users@xxxxxxxxxxxxxxxx
Subject: RE: [ldm-users] XFS small file performance and prevent Linux OOM from 
killing LDM

Hello Gilbert,

I am unsure what you are asking me.  This option is only necessary for XFS and 
EXT3/4 should not need it.  When you install Centos, you should have the option 
to select the filesystem type when creating the filesystems.
There isn't likely a means to set this on XFS created filesystems from the 
system installer. This setting for XFS will be the default on new centos
7.3 installs, so you won't have to worry about it there once 7.3 gets released.

daryl

On Mon, 10 Oct 2016, Gilbert Sebenste wrote:

Thanks, Daryl. Do you have the option to use EXT3 on install? I
haven't tried CentOS 7 yet.

Gilbert

-----Original Message-----
From: ldm-users-bounces@xxxxxxxxxxxxxxxx
[mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of daryl
herzmann
Sent: Tuesday, September 27, 2016 6:37 AM
To: ldm-users@xxxxxxxxxxxxxxxx
Subject: [ldm-users] XFS small file performance and prevent Linux OOM
from killing LDM

Greetings,

I'd like to share two Linux admin tips that may help you with your admining of 
LDM.  If you don't use the XFS filesystem nor have experienced the joy of 
Linux's Out of Memory (OOM) killer, you can safely skip this message.

XFS Small File Performance
==========================

Please, I don't wish to start a war regarding which filesystem is the best 
here...  If you have used XFS (now default filesystem in RHEL7) in the past, 
you may have suffered from very poor performance with IO related to small 
files.  For me and LDM, this would rear its very ugly head when I wished to 
`ldmadmin scour` the /data/ folder.  It would take 4+ hours to scour out a days 
worth of NEXRAD III files.  If you looked at output like sysstat, you would see 
the process at 100% iowait.

I created a thread about this on the redhat community forums[1] and was kindly 
responded to by one of the XFS developers, Eric Sandeen.  He wrote the 
following:

    This is because your xfs filesystem does not store the filetype in the
    directory, and so every inode in the tree must be stat'd (read) to
    determine the filetype when you use the "-type f" qualifier. This is
    much slower than just reading directory information. In RHEL7.3,
    mkfs.xfs will enable filetypes by default. You can do so today with
    "mkfs.xfs -n ftype=1".

So what he is saying is that you have to reformat your filesystem to take 
advantage of this setting.

So I did some testing and now `ldmadmin scour` takes only 4 minutes to 
transverse the NEXRAD III directory tree!

Linux OOM Killer
================

So when your Linux system starts running dangerously low on system memory, "it is 
the job of the linux 'oom killer' to sacrifice one or more processes in order to free up 
memory for the system when all else fails"[2].  Over the years, on heavily loaded 
systems I would see the `ldmd` process get killed as its memory footprint would be much 
larger than other processes running at the time.  Of course, having ldmd get killed by 
the system is not cool!

So there is a means to set a "score" on each Linux process to inform
the oom killer about how it should prioritizing the killing.  For
RHEL/centos
6+7, this can be done by `echo -1000 > /proc/$PID/oom_score_adj`.  For
some other Linux flavours, the score should be -17 and the proc file is 
oom_adj.  Google is your friend!

A simple cron script like so will set this value for ldmd
automatically each hour.  (This is all on one line...)

$ cat /etc/cron.d/oom_disable
1 * * * * root pgrep -f "ldmd" | while read PID; do echo -1000 >
/proc/$PID/oom_score_adj; done

Of course, this solution would have a small window of time between a ldm 
restart and the top of the next hour whereby the score would not be set.
There are likely more robust solutions here I am blissfully ignorant of.

later,
daryl

[1] https://access.redhat.com/discussions/476563
[2] https://linux-mm.org/OOM_Killer

--
/**
 * daryl herzmann
 * Systems Analyst III -- Iowa Environmental Mesonet
 * https://mesonet.agron.iastate.edu
 */

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are recorded in 
the Unidata inquiry tracking system and made publicly available through the 
web.  Users who post to any of the lists we maintain are reminded to remove any 
personal information that they do not want to be made public.


ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:
http://www.unidata.ucar.edu/mailing_lists/


--
/**
 * daryl herzmann
 * Systems Analyst III -- Iowa Environmental Mesonet
 * https://mesonet.agron.iastate.edu
 */


--
/**
 * daryl herzmann
 * Systems Analyst III -- Iowa Environmental Mesonet
 * https://mesonet.agron.iastate.edu
 */



  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: