[IDD #TTR-495389]: Slow access to local datasets through ADDE

["Unidata McIDAS Support" <support-mcidas@xxxxxxxxxxxxxxxx>]

Hi Samuel,

re:
> I started changing pqact.conf to try store the images in their own directory based on
> day, however, this may not make a lot of sense for images that only update a few times
> a day like Antarctic IR images.

Since it is easy to specify a regular expression that will match files in multiple directories
in the DSSERVE DIRFILE= keyword, I suggest that it is always a good idea to have some component
of the storage directory tree be date.  For instance, if you are storing the Antarctic IR
images in a directory tree that looks like:

/data/ldm/mcidas/images/sat/<date>/ANTARCTIC/IR/...

Then you can create an ADDE dataset that accesses the images from a single day:

DSSERVE ADD RTIMAGES/ANTARCTIC AREA DIRFILE=/data/ldm/mcidas/images/sat/\CURDAY/ANTARCTIC/IR/* "Antarctic IR Composite

or an ADDE dataset that access the images from all of the days you are keeping online:

DSSERVE ADD RTIMAGES/ANTARCTIC AREA DIRFILE=/data/ldm/mcidas/images/sat/*/ANTARCTIC/IR/* "Antarctic IR Composite

If you decided to keep 7 days of Antarctic IR images online, the dataset would be composed of 
56 images.


> I tried dividing up our RTIMAGES/ANTARCTIC images, but IDV no reports "No data available for the selection"
> when trying to access RTIMAGES/Antarctic IR Composite.  There is imagery in the child directory where
> the IR images used to be, and the dsserve.k command seemed to work.  I'm at a loss as to why it didn't work.
> 
> Here is a log of what happened:
> 
> -bash-2.05b$ dsserve.k ADD RTIMAGES/ANTARCTIC AREA TYPE=IMAGE DIRFILE=/data/ldm
> /mcidas/images/sat/ANTARCTIC/4km/IR/\\CURDAY/IR_* \"Antarctic IR Composite

In this dsserve.k invocation, you did not escape the '*'.  The invocation should
have been:

<as 'mcidas'>
cd $MCDATA
dsserve.k ADD RTIMAGES/ANTARCTIC AREA TYPE=IMAGE DIRFILE=/data/ldm/mcidas/images/sat/ANTARCTIC/4km/IR/\\CURDAY/IR_\* "Antarctic IR Composite

> Group/Descriptor         Type  Format & Range     RT Comment
> ------------------------ ----- ------------------ -- --------------------
 ...
> RTIMAGES/ANTARCTIC       IMAGE AREA                  Antarctic IR Composite
> DIRFILE=/data/ldm/mcidas/images/sat/ANTARCTIC/4km/IR/\CURDAY/IR_*
> dsserve.k: done

Hmm... This definition looks OK to me.  

> -bash-2.05b$ ls /data/ldm/mcidas/images/sat/ANTARCTIC/4km/IR/20080206/IR_200802
> 06_1500 -l
> -rw-rw-r--    1 ldm      ldm        142425 Feb  6 12:44 /data/ldm/mcidas/images/
> sat/ANTARCTIC/4km/IR/20080206/IR_20080206_1500

This looks like it should have worked.  What is the result if you try to access
the RTIMAGES/ANTARCTIC images from McIDAS?

try:

<login as 'mcidas'>
cd $MCDATA
dataloc.k LIST RTIMAGES      <- verify that your McIDAS session will look for the
                                RTIMAGES dataset on shu either as a LOCAL-DATA
                                dataset or by pointing to the remote ADDE interface
                                on shu.cup.edu (I am assuming that you are serving
                                the data from shu.cup.edu; if not point to the machine
                                that is doing the serving)
imglist.k RTIMAGES/ANTARCTIC

What is the result?

> P.S. Is there a way to save these changes once we're happy with them, or are they
> saved automaticaly on reboot?

The ADDE dataset definitions created by DSSERVE are stored in the disk file $MCDATA/RESOLV.SRV,
so the definitions are preserved across McIDAS invocations and system reboots.

By the way, it would help me in troubleshooting if I could either get SSH access to
shu as 'mcidas' again (I understand about there being login attempts by many hackers,
but shu's firewall can be configured to allow SSH access by machines in a single domain or,
if needed, by a single machine), and/or to open up access to the ADDE server on
shu.cup.edu so I can run McIDAS commands from my office.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
support@xxxxxxxxxxxxxxxx                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TTR-495389
Department: Support McIDAS
Priority: Normal
Status: Closed