[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #JEK-162128]: Using scour with grbfile.sh

Hi Carol,

re:
> Greg Stossmeister implemented your grbfile.sh script to archive GOES data.
> I then wrote a separate script to archive the data to a location on our
> HPSS.

Progress! :-)

re:
> I noticed today that the $1/Channel$4/$ccyymmdd directories are not being
> scrubbed/scoured.

It is the user's job to setup scouring of any/all data processed
to disk by the LDM.

re:
> From what I can tell, there isn't any scrub or scour within the grbfile.sh.

Correct, except that the number of files kept in each 'current' directory
is set by the KEEP variable in grbfile.sh.

re:
> Do you have any suggestions for how to efficiently setup a scour for this
> GOES data? Can I use wildcards like this?
> 
> /data/ldm/pub/native/satellite/GOES/GRB16/ABI/*/*7*.nc
> /data/ldm/pub/native/satellite/GOES/GRB16/GLM/LCFA/*7*.nc

Our approach to data scouring is to keep a certain number of
days of data on disk.  For the most part, the scour utility
that is part of the LDM can be used to do the job, but its
shortcoming is that it does not delete empty directories.  A
long time ago, I wrote a couple of Tcl-based scouring tools
that do what we want to do -- delete files and remove empty
directories.  We have also used C-shell scripts to do the same
thing, and we also developed a Perl scouring module that is
configured in the same way that the LDM scour utility uses.
We have seen that sites often like to roll their own scouring
routines, so we have not tried to package our adhoc scouring
solutions with, say, the LDM.

I can send you my Tcl-based scouring routines if you like
(or copy them over to the ~ldm/util directory on typhoon).
Please let me know if you are interested.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: JEK-162128
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.