[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20000317: NIDS and NOWrad setup (cont.)

>From: "Karli Lopez (McIDAS)" <address@hidden>
>Organization: University of Puerto Rico
>Keywords: 200002010754.AAA03412 NIDS NOWRAD ADDE NIDS.CFG

Karli,

re: 'WSI<tab>...' line in pqact.conf
>Oh ok, that's what i'd done (merge it into a single line) but I missed placing
>the first tab before the "FILE" entry.
>I had:
>----------------
>WSI<tab>...<newline>
>FILE<tab>...
>----------------
>instead of:
>----------------
>WSI<tab>...<newline>
><tab>FILE<tab>....
>----------------
>it is now "syntactically correct"!

Excellent.

>Now do I need the directories to be /data/.../directoryname/NIDS, or just /dat
> a/.../directoryname where "/data/directoryname" is the directory where the
>files are located (in out case it was set up orginally as "/unidata/ldm/data/m
> cidas/")

>my guess is that it's just the directory name and that NIDS is not
>necessary, but I want to be sure.

You are right.  The location information in pqact.conf is where you
want to put the data.  This is up to you.  The DIRMASK entry in the
McIDAS ADDE configuration file for NIDS data,
~mcidas/workdata/NIDS.CFG, will need to be set to match whatever
directory location you choose.

The concept here is that the NIDS ADDE server needs to be able to find
where the NIDS data files are and how they are named.  The regular
expression that is the value for the DIRMASK= parameter in NIDS.CFG
describes where the files are located.  The regular expression for the
FILEMASK= parameter in NIDS.CFG describes how the files are named.

Perhaps a little "graphical" example of how this would look might be
useful.

Remember, these are only examples; you have to setup your
system according to the locations you choose for the data files:

pqact.conf entry says:

WSI     
^NEX/(.*)/(.*)/([1-2][0-9])([0-9][0-9])([0-1][0-9])([0-3][0-9])([0-2][0-9])([0-6][0-9])
        FILE    data/nexrad/\1/\2/\2_\3\4\5\6_\7\8

In this example, data/nexrad/... refers to the ~ldm/data/nexrad... directory.
Remember that ~ldm/data is typically a link to some other directory. For
the purpose of illustration, let us assume that ~ldm/data is a link to
/data.

The files will be stored in directories that look like:

cd ~ldm/data/nexrad
pwd
/data/nexrad
ls
ABX  CYS  ENX  FTG  GRK  OKX  PUX

As you can see, we are getting NIDS data for several sites: ABS, CYS, ENX,
FTG, GRK, OKX, and PUX.  These directory names are the NIDS station ID names.

cd FTG
ls
BREF1   BREF4   CREF16  LREF3   PRE3    PRET    SRMV1   TOPS    VEL2    VIL
BREF2   BRLR1   LREF1   PRE1    PRE3X   PRETX   SRMV2   VAD     VEL3
BREF3   CREF    LREF2   PRE1X   PREA    RESP    STAT    VEL1    VEL4

There is one directory for each of the NIDS product types: BREF1, BREF2, etc.

cd BREF1
ls
BREF1_20000318_1115  BREF1_20000318_1320  BREF1_20000318_1525
BREF1_20000318_1125  BREF1_20000318_1330  BREF1_20000318_1530
BREF1_20000318_1130  BREF1_20000318_1335  BREF1_20000318_1540
BREF1_20000318_1135  BREF1_20000318_1340  BREF1_20000318_1545
BREF1_20000318_1140  BREF1_20000318_1345  BREF1_20000318_1550
BREF1_20000318_1145  BREF1_20000318_1350  BREF1_20000318_1555
BREF1_20000318_1155  BREF1_20000318_1355  BREF1_20000318_1600
BREF1_20000318_1200  BREF1_20000318_1405  BREF1_20000318_1605
BREF1_20000318_1205  BREF1_20000318_1410  BREF1_20000318_1615
BREF1_20000318_1210  BREF1_20000318_1415  BREF1_20000318_1620
 ...

(these are the actual data files.  They are named BREF_CCYYMMDD_HHMM
where:

  CCYYMMDD_HHMM
  ^ ^ ^ ^  ^ ^__ minute of the hour
  | | | |  |____ hour of the day
  | | | |_______ day of the month
  | | |_________ month of the year
  | |___________ year of the century
  |_____________ century

BREF1 is the product type.  In the way I setup the ADDE NIDS data set
(in ~mcidas/data/DSSERVE.BAT), it is also part of the data set name:

  RTNIDS/BREF1             <- realtime NIDS Base Reflectivity at tilt 1 data
  RTNIDS/BREF2             <- realtime NIDS Base Reflectivity at tilt 2 data
  RTNIDS/BREF3             <- realtime NIDS Base Reflectivity at tilt 3 data
  RTNIDS/BREF4             <- realtime NIDS Base Reflectivity at tilt 4 data
  RTNIDS/LREF1             <- realtime NIDS Layer Reflectivity at level 1 data
  RTNIDS/LREF2             <- realtime NIDS Layer Reflectivity at level 2 data
  RTNIDS/LREF3             <- realtime NIDS Layer Reflectivity at level 3 data
  RTNIDS/CREF              <- realtime NIDS Composite Reflectivity data

and so on.

The NIDS.CFG file will contain entries that look like:

DIRMASK=/data/nexrad/\ID/\TYPE
FILEMASK=\TYPE_*
IDMASK=*

The "replaceable" token '\ID' represents the value of the ID= keyword
that will be passed by the IMGDISP command that will load images.  The
\TYPE "replaceable" is the NIDS product type (e.g., BREF1, BREF2,
etc.).  The "replaceable" concept is not part of regular expressions;
it is a concept that I added to make setting up the configuration file
easier.

A full regular expression for a NIDS data file then becomes the
concatenation of the DIRMASK and FILEMASK values:

/data/nexrad/\ID/\TYPE/\TYPE_*

A load request to the server that looks like:

IMGDISP RTNIDS/BREF1 ID=FTG

results in the regular expression being expanded to:

/data/nexrad/FTG/BREF1/BREF1_*

This will match all of the data files in /data/nexrad/FTG/BREF1 that
begin BREF1_.

Similarily, an image load that looks like:

IMGDISP RTINDS/CREF ID=PUB

will result in the regular expression:

/data/nexrad/PUB/CREF/CREF_*

This flexibility allows one to actually put all of the NIDS data files
into a single directory (like for a case study).  For instance,
suppose that you had a directory named:

/var/data/sanjuan/case1

that had all NIDS products for the San Juan, Puerto Rico NEXRAD
(station ID JUA _I think_).  A directory listing might look like:

cd /var/data/sanjuan/case1
ls
BREF1_20000212_1530  BREF1_20000212_1535 ...
BREF2_20000212_1530  BREF2_20000212_1535 ...
BREF3_20000212_1530  BREF3_20000212_1535 ...
BREF4_20000212_1530  BREF4_20000212_1535 ...
LREF1_20000212_1530  LREF1_20000212_1535 ...
LREF2_20000212_1530  LREF2_20000212_1535 ...
 ...

The DIRMASK= and FILEMASK= parameters in NIDS.CFG would look like:

DIRMASK=/var/data/sanjuan/case1/
FILEMASK=\TYPE_*

Notice that since there is only data for JUA in the directory that
I do not need to use the \ID replaceable.

Following on with the same kind of example, suppose I have all NIDS
products for FTP (Denver, CO) and PUB (Pueblo, CO) in the same
directory, /var/data/colorado.  A directory listing might look like:

cd /var/data/colorado
ls
FTG-BREF1_20000212_1530  FTG-BREF1_20000212_1535 ...
FTG-BREF2_20000212_1530  FTG-BREF2_20000212_1535 ...
FTG-BREF3_20000212_1530  FTG-BREF3_20000212_1535 ...
FTG-BREF4_20000212_1530  FTG-BREF4_20000212_1535 ...
FTG-LREF1_20000212_1530  FTG-LREF1_20000212_1535 ...
FTG-LREF2_20000212_1530  FTG-LREF2_20000212_1535 ...
PUB-BREF1_20000212_1530  PUB-BREF1_20000212_1535 ...
PUB-BREF2_20000212_1530  PUB-BREF2_20000212_1535 ...
PUB-BREF3_20000212_1530  PUB-BREF3_20000212_1535 ...
PUB-BREF4_20000212_1530  PUB-BREF4_20000212_1535 ...
PUB-LREF1_20000212_1530  PUB-LREF1_20000212_1535 ...
PUB-LREF2_20000212_1530  PUB-LREF2_20000212_1535 ...
 ...

The DIRMASK= and FILEMASK= parameters would look like:

DIRMASK=/var/data/colorado/
FILEMASK=\ID-\TYPE_*

As you can see, this setup can be pretty flexible.

>So is McIDAS Area conversion turned off after i uncomment the original WSI
>lines from pqact.conf? I want to have both things active.

No.  In order to turn off conversion to AREA files, you would have to comment
out all of the WSI lines that call for the invocation of the nids2area
decoder.

>many Thanks,

You are welcome.

By the way.  I have been meaning to talk with you about your ADDE setup.
It appears that the executables 'compress' and 'uncompress' can not
be found by the ADDE routines on your system.  A quick look at our
IRIX machines here at Unidata show:

uname -a
IRIX chevy 6.5 01200531 IP22

which compress uncompress
/usr/ucb/compress
/usr/ucb/uncompress

If compress and uncompress are not located in /usr/ucb on your system,
then the ADDE servers can not send/receive compressed data.

Compressed ADDE data transfers are controlled by an environment variable
named MCCOMPRESS that has to be set for each user that is going to
run McIDAS-X.  If MCCOMPRESS is not defined, it is assumed to mean
that data is to be transferred in an uncompressed format.  This may
be the reason that your IMGDISP RTIMAGES/GE... invocations were so
slow when your DATALOC for RTIMAGES was pointing back here to Unidata.
MCCOMPRESS must be set _before_ the McIDAS-X session is started.

Here is a sequence that will demonstrate how much transferring
compressed data helps:

<login as 'mcidas'>
unsetenv MCCOMPRESS         (assumes you are using the C shell; adjust the
                             syntax if you are using the Bourne or Korn
                             shells)
cd workdata
mcidas

DATALOC ADD RTIMAGES ADDE.UNIDATA.UCAR.EDU
IMGDISP RTIMAGES/GE-IR STA=TJSU EU=IMAGE SF=YES REFRESH='EG;MAP H 1'

(try to get a feel for how long this took to complete)

EXIT

setenv MCCOMPRESS TRUE

mcidas

IMGDISP RTIMAGES/GE-IR STA=TJSU EU=IMAGE SF=YES REFRESH='EG;MAP H 1'

(compare how long this took with the uncompressed transfer)

DATALOC ADD RTIMAGES BREEZE.UPRM.EDU

IMGDISP RTIMAGES/GE-IR STA=TJSU EU=IMAGE SF=YES REFRESH='EG;MAP H 1'

(this invocation should fail until 'compress' and 'uncompress' and
can be found in the environment in which the remote ADDE server on
breeze runs.  The remote ADDE server routine that looks for 'compress'
and 'uncompress' searches the following directories (in this order):

/bin
/usr/ucb
/usr/bin
/usr/bsd

If 'compress' and 'uncompress' are in different directories on your
system, then you should make a link from where they are to the /usr/ucb
directory (has to be done as 'root').  This should get things working.

I hope that this email was not too long to digest.  I try to explain
things in detail since these email exchanges go into an inquiry tracking
system that all users can search and, hopefully, learn from.

Tom