[ldm-users] 20200424: Re: ldm data dir question

Hi Jack,

On 4/24/20 1:20 PM, Jack Snodgrass wrote:
just to confirm... find and rm on the data dir won't mess up / confuse the ldm queue stuff?

It should not.  But, more to the poing, the find invocation you
indicated that you were running was limited to the ~ldm/var/data
directory and your LDM queue should be in the ~ldm/var/queues
directory.

By the way, your machine becoming unreachable is not explained by
the /home file system filling.  It is possible the root file
system, '/', is also full.

What is the output from:

df -h

Cheers,

Tom


On 4/24/20 2:00 PM, Evan Breznyik wrote:
That was going to be my next suggestion.  When we had 1M+ files on disk, that was too many arguments for any standard command, and scour is...well, slow.

The loop/delete is how I brought that down.

XFS with inode64 is also what we use...so I know that to work.  We also just upped the frequency of scour, but we're also running on NVME/SSD with 24+ core processors...so we can afford to thrash the disk.  I realize people on spinning disks may not be open to this same option.

On Fri, Apr 24, 2020 at 11:48 AM Jack Snodgrass <jack@xxxxxxxxxxxxxx <mailto:jack@xxxxxxxxxxxxxx>> wrote:

    file system is xfs.

    /dev/sde5 on /home type xfs (rw,relatime,attr2,inode64,noquota)

    df -i /home
    Filesystem        Inodes  IUsed     IFree IUse% Mounted on
    /dev/sde5      188351872 169213 188182659    1% /home


    I have removed about 150K files from /home/.... /data  but letting
    the

    find /home/ldm/var/data -type f -mtime +1 -print | xargs rm -f

    command keep running... I'll try and restart ldm when that finishes.

    - jack



    On 4/24/20 1:41 PM, Evan Breznyik wrote:
    What filesystem is this on...vis a vis how are the inodes
    looking?  I hit a corner case with this a few years ago where we
    were saving 100K+ files per day and we simply ran out of inodes
    on ext4...but I am just thinking aloud.



    On Fri, Apr 24, 2020 at 11:32 AM Jack Snodgrass
    <jack@xxxxxxxxxxxxxx <mailto:jack@xxxxxxxxxxxxxx>> wrote:

        having issues with our server ( centos7 ) that runs ldm...
        locking up. It has happened 2 times in the last 3 weeks or so.
        The server is pingable... so it's not totally dead.. but you
        can't get a local or remote console to start. can't figure
        out if it is out of memory or file handles or what.... it's
        like a ghost of itself.

        After rebooting... the  /home/ldm/var/data/ has around
        350,000 files in it.  I am not sure if that is 'ok' or a bit
        extra.

        We are running a

        ldmadmin scour

        command... via cron but I don't know what that is doing
        exactly or it it's doing much.

        when I try and restart ldm it says:

        Checking the product-queue...
        The writer-counter of the product-queue isn't zero.  Either a
        process
        has the product-queue open for writing or the queue might be
        corrupt.
        Terminate the process and recheck or use
            pqcat -l- -s -q /home/ldm/var/queues/ldm.pq && pqcheck -F -q
            /home/ldm/var/queues/ldm.pq
        to validate the queue and set the writer-counter to zero.
        LDM not started


        In the past.... during testing and what not.. I've been able
        to run:
        pqcat -l- -s -q /home/ldm/var/queues/ldm.pq && pqcheck -F
        -q/home/ldm/var/queues/ldm.pq

        and ldm would start after that. This time.. with the 350K
        files or so.. that pqcat stuff fails.

        I am deleting older ( than a day ) files from the
        /home/ldm/var/data/ direcory... going to see if

        pqcat -l- -s -q /home/ldm/var/queues/ldm.pq && pqcheck -F
        -q/home/ldm/var/queues/ldm.pq


        will work or if I have to rm -rf /home/ldm/var/data/ and
        start a new q.


        If  ldmadmin scour does not let us remove enough files from
        /home/ldm/var/data/ can I use find and rm to remove files or
        do they have to be removed using ldm to keep and queses or
        indexes  in sync?

        - jack

-- *jack* - Southlake Texas - http://mylinuxguy.net
        <http://mylinuxguy.net/> - *817-601-7338*
        _______________________________________________
        NOTE: All exchanges posted to Unidata maintained email lists are
        recorded in the Unidata inquiry tracking system and made publicly
        available through the web.  Users who post to any of the lists we
        maintain are reminded to remove any personal information that
        they
        do not want to be made public.


        ldm-users mailing list
        ldm-users@xxxxxxxxxxxxxxxx <mailto:ldm-users@xxxxxxxxxxxxxxxx>
        For list information or to unsubscribe,  visit:
        https://www.unidata.ucar.edu/mailing_lists/



-- *jack* - Southlake Texas - http://mylinuxguy.net
    <http://mylinuxguy.net/> - *817-601-7338*
    _______________________________________________
    NOTE: All exchanges posted to Unidata maintained email lists are
    recorded in the Unidata inquiry tracking system and made publicly
    available through the web.  Users who post to any of the lists we
    maintain are reminded to remove any personal information that they
    do not want to be made public.


    ldm-users mailing list
    ldm-users@xxxxxxxxxxxxxxxx <mailto:ldm-users@xxxxxxxxxxxxxxxx>
    For list information or to unsubscribe,  visit:
    https://www.unidata.ucar.edu/mailing_lists/



--
*jack* - Southlake Texas - http://mylinuxguy.net <http://mylinuxguy.net/> - *817-601-7338*

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
https://www.unidata.ucar.edu/mailing_lists/


--
+----------------------------------------------------------------------+
* Tom Yoksas                                      UCAR Unidata Program *
* (303) 497-8642 (last resort)                           P.O. Box 3000 *
* yoksas@xxxxxxxx                                    Boulder, CO 80307 *
* Unidata WWW Service                     http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------------+


  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: