[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20021112: refining how many days of McIDAS data are kept online (cont.)

>From: William C Klein <address@hidden>
>Organization: Valparaiso
>Keywords: 200208191558.g7JFw6K19855 McIDAS-X mcscour.sh

Bill,

>I never rec. any information on this matter.  Please confirm.

I looked through the attached email exchanges between us, and I don't
see a pending question.  What have you been waiting on?

My recollection is that there must be a different scouring process on
aeolus that is removing the McIDAS data files leaving less than the
number you want (which I believe, was 9).  One possibility for this
would be the scouring done by the LDM scour routine, but the
~ldm/etc/scour.conf file on your system does not show that the
directory containing McIDAS data is being scoured.

I just logged onto aeolus and see that you are now keeping at least
7 days of MD (POINT) data:

<login as 'mcidas'>
cd workdata
dmap.k MDXX

I see the same result when accessing aeolus through your remote ADDE
server from a McIDAS session here at the UPC:

DATALOC ADD RTPTSRC AEOLUS.VALPO.EDU

Group Name                    Server IP Address
--------------------         ----------------------------------------
RTPTSRC                      AEOLUS.VALPO.EDU

<LOCAL-DATA> indicates that data will be accessed from the local data directory.
DATALOC -- done

PTLIST RTPTSRC/SFCHOURLY FORM=FILE ALL
Pos      Description                        Schema  NRows NCols Proj# Created 
DataDate
------   --------------------------------   ------  ----- ----- ----- ------- 
--------
     1   SAO/METAR data for   07 NOV 2002   ISFC       72  6500     0 2002311 
2002311
     2   SAO/METAR data for   08 NOV 2002   ISFC       72  6500     0 2002311 
2002312
     3   SAO/METAR data for   09 NOV 2002   ISFC       72  6500     0 2002312 
2002313
     4   SAO/METAR data for   10 NOV 2002   ISFC       72  6500     0 2002313 
2002314
     5   SAO/METAR data for   11 NOV 2002   ISFC       72  6500     0 2002315 
2002315
     6   SAO/METAR data for   12 NOV 2002   ISFC       72  6500     0 2002316 
2002316
    10   SAO/METAR data for   06 NOV 2002   ISFC       72  6500     0 2002310 
2002310
PTLIST: Done


Since I can't logon to aeolus as 'ldm', I can't get 'ldm's crontab listing.
This will tell us which copy of mcscour.sh is being run to scour data,
and the contents of that file will tell us how many days of data you
have configured your system to keep.

Again, I am not sure of what additional information you were waiting to
get from me.

Tom

>On Tue, 8 Oct 2002, William C Klein wrote:
>
>> Tom,
>>
>> I think that the mcscour.sh is being run from a mounted directory and
>> that's why we are not getting the full amount of data.
>>
>> Bill
>>
>> On Tue, 1 Oct 2002, Unidata Support wrote:
>>
>> > >From: William C Klein <address@hidden>
>> > >Organization: Valparaiso
>> > >Keywords: 200208191558.g7JFw6K19855 McIDAS-X mcscour.sh
>> >
>> > Bill,
>> >
>> > >Here is the snibbit of the mcscour.sh that you were talking about from my
>> > >machine:
>> > >
>> > >MCPATH=$MCPATH PATH=$PATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH mcenv << EOF
>> > >
>> > >qrtmdg.k GRID 5001 6000 1
>> > >doqtl.k  1  70 9
>> > >doqtl.k 71  80 9
>> > >doqtl.k 81  90 9
>> > >doqtl.k 91 100 9
>> > >delwxt.k 1 10
>> > >igu.k DEL 132
>> > >lwu.k DEL VIRT9001
>> > >lwu.k DEL VIRT9002
>> > >lwu.k DEL ROUTEPP.LOG
>> > >exit
>> > >
>> > >EOF
>> > >
>> > ># Done
>> > >exit 0
>> >
>> > If this really is the mcscour.sh file that is being used to scour McIDAS
>> > data, you should be keeping 9 days of POINT source files online.  Since
>> > I can point at your remote ADDE server (on aeolus.valpo.edu) and only
>> > see two days worth of data, it is most likely that a different copy of
>> > mcscour.sh is being used to scour the McIDAS data.  It is even possible
>> > that you have two different invocations of mcscour.sh running from cron.
>> > Perhaps one is running from the 'mcidas' account and a different one
>> > is running from the 'ldm' account?  If this is the case, and the second
>> > mcscour.sh is setup for the default number of days to keep (2), then
>> > I could easily understand how there would be a discrepency between the
>> > listing above and the number of files that we see on your machine.
>> >
>> > >[ aeolus : mcidas : ~/bin ]
>> > >[ 9 ] >
>> > >
>> > >It looks as though it's already setup for 9 days of data.  Should there b
> e
>> > >a change elsewhere?
>> >
>> > No, not if you are only scouring using mcscour.sh.
>> >
>> > >Like in:
>> > >
>> > >[ aeolus : ldm : ~/etc ] ???  There is a scour.conf.
>> >
>> > I strongly recommend that the LDM scouring mechanism _NOT_ be used to scou
> r
>> > McIDAS data.  The main reason (unless one is really careful to setup thing
> s
>> > up correctly) is that the directory containing McIDAS-XCD decoded files
>> > should contain files that do not get updated (i.e., get old).  The LDM
>> > scouring approach is to delete files from directories that are older
>> > than a certain number of days.  What could happen if LDM scouring is
>> > setup to scour McIDAS-XCD produced files is that these files (SCHEMA, in
>> > particular) would get deleted after they got to be 'n' days old.  This
>> > would be _bad_.  If one uses the mcscour.sh approach, this will not
>> > happen.
>> >
>> > >Does this also limit the # of days that are being held?
>> >
>> > If you are scouring the directory that McIDAS-XCD files are being written
>> > into, then yes, this would also limit the number of days being held.
>> >
>> > I recommend that you check the following:
>> >
>> > 1) make sure that mcscour.sh is only being run from one account.  I recomm
> end
>> >    running it from the 'ldm' account.
>> >
>> > 2) make sure that the single copy of mcscour.sh that is being run is
>> >    configured the way you want (like the listing above)
>> >
>> > 3) make sure that the LDM scouring is _not_ setup to scour the directory
>> >    containing McIDAS-XCD produced data files.
>> >
>> > Once you have only one copy of mcscour.sh being run by cron, you will
>> > start keeping more days of data on line.
>> >
>> > >Thanks,
>> >
>> > Please keep me up to date with your problem.  I will be in and out
>> > for the next couple of days as I am attending the GOES Users Conference
>> > here in Boulder.
>> >
>> > Tom
>> > **************************************************************************
>> > Unidata User Support                                    UCAR Unidata Progr
>> > (303)497-8643                                                  P.O. Box 30
>> > address@hidden                                   Boulder, CO 803
>> > --------------------------------------------------------------------------
>> > Unidata WWW Service                        http://www.unidata.ucar.edu/   
>> > **************************************************************************
>> >
>>
>> ---------------------
>> William C. Klein
>> Webmaster
>> Valparaiso University
>> ---------------------
>>
>>
>
>---------------------
>William C. Klein
>Webmaster
>Valparaiso University
>---------------------
>