It's my opinion, untested, that xfs is slightly slower for this sort of
operation than zfs, but ANY of the journaled file systems should be
similar. Karen's solution is one I've not looked at, but sounds elegant.
gc
On Wed, Oct 28, 2015 at 2:43 PM, Karen Cooper - NOAA Affiliate <
karen.cooper@xxxxxxxx> wrote:
> We've mostly used it on ext2 and ext3. We have also used it on zfs.
>
> I don't think we have any xfs filesystems that we have run it on.
>
> On Wed, Oct 28, 2015 at 2:35 PM, Donna Cote <d-cote@xxxxxxxx> wrote:
>
>> Karen, the C program you
>> guys
>> wrote sound
>> s
>> really neat.
>> What kind of filesystem are you using? xfs? ext4? zfs?
>>
>> What I have heard is that a remove done on xfs can take a really long
>> time compared to a remove done on zfs.
>>
>> We switched the hard drives on our LDM storage server to zfs and find
>> that a scour every hour works well.
>>
>> We can't do the same on our EDEX / AWIPS II server and would just like
>> to get it running smoothly. It seems I'm in constant nanny mode with it.
>>
>> Donna
>>
>>
>> On Oct 28, 2015 11:45 AM, "Karen Cooper - NOAA Affiliate" <
>> karen.cooper@xxxxxxxx> wrote:
>>
>>> Here at NSSL we wrote a cleanup program because at times scour did not
>>> keep up with our data.
>>>
>>> To be fair, we were using scour to clean out not only our incoming data,
>>> but also a lot of products (MRMS) we were creating from that data.
>>>
>>> We had originally tried Gerry's suggested method of running scour more
>>> frequently, which worked as long is it did not "stack". Stacking meaning
>>> that a new scour would begin before the last was complete. When that
>>> happened they would typically continue to stack and spiral out of control.
>>>
>>> So we wrote our own that would do single recursive searches of our
>>> directories rather than having to do multiple passes on directories,
>>> looking for different patterns. It runs as a daemon, so when it is done,
>>> it sleeps for a configurable amount of time and then runs again.
>>>
>>> We could probably have just been more careful with crafting our patterns
>>> in scour.conf, but since we used it so heavily the C program seemed the way
>>> to go and it has worked very well for us.
>>>
>>> On Wed, Oct 28, 2015 at 11:31 AM, Smith, Neil R <n-smith2@xxxxxxxxxxxxx>
>>> wrote:
>>>
>>>> Good question - isn’t it the same I/O process, just a matter of when?
>>>>
>>>> I'm wondering if the programming overhead of a file modification time
>>>> parameter test conducted by the ‘find’ command makes it worth avoiding.
>>>>
>>>> Gerry Creager is suggesting that a solution is to just increase the
>>>> frequency of running a find command (scour). It’s worth a try, but I don’t
>>>> see how that escapes the convergence of time to eval delete qualifications
>>>> inode list expansion rate.
>>>>
>>>> But if that’s a working solution, then — in the limit — so is a file
>>>> delete action executed coincident with a new file write.
>>>>
>>>> I interpret Tom Yoksas as saying to fail over to a different file
>>>> system catalogue operation - removal of groups of inodes instead of removal
>>>> of each inode separately.
>>>>
>>>> Good idea. And the one Donna Cote settled on. Which will require me to
>>>> adjust all my friggin gempak GUI configs and scripted real-time graphics
>>>> product generation.
>>>>
>>>> Which I'm getting paid to do ...
>>>>
>>>> -Neil
>>>>
>>>> > On Oct 28, 2015, at 10:28 AM, Donna Cote <d-cote@xxxxxxxx> wrote:
>>>> >
>>>> > Neil, while it is possible to use the EXEC argument, given the
>>>> filename, to create the rm command line statment (on that filename minus x
>>>> number of days), I don't see how this would make less removes than you are
>>>> already doing.
>>>> >
>>>> > I would like to ask for a more io-friendly scour utility.
>>>> >
>>>> > Donna
>>>> > Texas A&M University
>>>> > The Academy
>>>> >
>>>> >
>>>> > On Wed, Oct 28, 2015 at 10:17 AM, Smith, Neil R <
>>>> n-smith2@xxxxxxxxxxxxx> wrote:
>>>> > Scouring 10 days of online, or even nearline, NIDS and NEXRCOMP being
>>>> held for a teaching environment is a big headache.
>>>> > Donna Cote has recently polled the community about this issue.
>>>> >
>>>> > At this file count, crawling thru the millions of inodes with a
>>>> scheduled find command will not keep up with the accumulation rate.
>>>> > (sounds like the intersection of two curves, beyond which there is no
>>>> return; would anyone care to derive that solution?)
>>>> >
>>>> > This leads me to wonder if one can construct a pqact entry that will
>>>> do the FILE or STDIOFILE action on the incoming product plus remove the
>>>> same product of the same date minus a desired day count. Or the remove
>>>> action, -exec, could be a separate pqact entry on the same product regular
>>>> expression minus the day count, just so it’s done at the time of the new
>>>> product arrival. Is there a clever way to manipulate the syntax of
>>>> temporal subexpression replacement described in the pqact man pages to
>>>> generate the desired filename to remove?
>>>> >
>>>> > Would that even solve the problem?
>>>> >
>>>> > Or is there a pqact solution? Are you just left with scripting OS
>>>> commands (find; eg. scour)?
>>>> >
>>>> > -Neil
>>>> >
>>>> > --
>>>> > Neil R. Smith, Senior IT Specialist II neils@xxxxxxxx
>>>> > Atmospheric Sciences, Texas A&M Univ. 979/845-6272
>>>> > --
>>>> >
>>>> > _______________________________________________
>>>> > ldm-users mailing list
>>>> > ldm-users@xxxxxxxxxxxxxxxx
>>>> > For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>> >
>>>>
>>>> _______________________________________________
>>>> ldm-users mailing list
>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
>>>
>>>
>>> --
>>> “The presence of those seeking the truth is infinitely to be preferred
>>> to the presence of those who think they’ve found it."
>>> -- Terry Prachett
>>>
>>> -------------------------------------------
>>> Karen.Cooper@xxxxxxxx
>>>
>>> Phone#: 405-325-6456
>>> Cell: 405-834-8559
>>> National Severe Storms Laboratory
>>>
>>>
>
>
> --
> “The presence of those seeking the truth is infinitely to be preferred to
> the presence of those who think they’ve found it."
> -- Terry Prachett
>
> -------------------------------------------
> Karen.Cooper@xxxxxxxx
>
> Phone#: 405-325-6456
> Cell: 405-834-8559
> National Severe Storms Laboratory
>
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
--
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
“Big whorls have little whorls,
That feed on their velocity;
And little whorls have lesser whorls,
And so on to viscosity.”
Lewis Fry Richardson (1881-1953)
