[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20000106: McIDAS remote ADDE server (cont.)

>From: "Thomas L. Mote" <address@hidden>
>Organization: University of Georgia
>Keywords: 200001042125.OAA01383 McIDAS-XCD setup

Tom,

re: you using NIS+
>You are correct! I am running NIS+. I added the files entry 
>under services in my /etc/nsswitch.conf file. After 
>rebooting, I was able to do a a telnet cacimbo.ggy.uga.edu 
>500 and get it to respond as it should.

Excellent!   I just pointed my McIDAS session at your machine
and verified that I can talk to your ADDE remote server:

DATALOC ADD RTIMAGES cacimbo.ggy.uga.edu
DSINFO IMAGE RTIMAGES
        Dataset Names of Type: IMAGE in Group: RTIMAGES

Name         NumPos   Content
------------ ------   --------------------------------------
ANTARCTIC       10    Antarctic IR Composite
EDFLOATER-I     10    Educational Floater
EDFLOATER-II    10    Educational Floater II
GE-IR           10    GOES-East North America IR
GE-IRTOPO       10    GOES-East IR/TOPO Composite
GE-VIS          10    GOES-East North America VIS
GE-VISTOPO      10    GOES-East VIS/TOPO Composite
GE-WV           10    GOES-East North America H2O
GEW-IR          10    GOES-East/West IR Composite
GEW-IRTOPO      10    GOES-East/West IR/TOPO Composite
GEW-VIS         10    GOES-East/West VIS Composite
GEW-VISTOPO     10    GOES-East/West VIS/TOPO Composite
GEW-WV          10    GOES-East/West H2O Composite
GW-IR           10    GOES-West Western US IR
GW-IRTOPO       10    GOES-West IR/TOPO Composite
GW-VIS          10    GOES-West Western US VIS
GW-VISTOPO      10    GOES-West VIS/TOPO Composite
GW-WV           10    GOES-West Western US H2O
MDR             10    Manually Digitized Radar
MDRTOPO         10    MDR/TOPO Composite
MOLL-IR         10    Mollweide Composite IR
MOLL-IRTOPO     10    Mollweide IR/TOPO Composite
MOLL-WV         10    Mollweide Composite H2O
RESFLOATER      10    Research Floater

DSINFO -- done

I also verified that I can see images on your system:

IMGLIST RTIMAGES/GE-IR.ALL
Image file directory listing for:RTIMAGES/GE-IR
 Pos Satellite/         Date       Time      Center   Band(s)
     sensor                                 Lat  Lon
 --- -------------  ------------  --------  ---- ---- ------------
   1  G-8 IMG        6 JAN 00006  12:15:00    23   71 4
   2  G-8 IMG        6 JAN 00006  13:15:00    23   71 4
   3  G-8 IMG        6 JAN 00006  14:15:00    23   71 4
   4  G-8 IMG        6 JAN 00006  15:15:00    23   71 4
   5  G-8 IMG        6 JAN 00006  16:15:00    23   71 4
   6  G-8 IMG        6 JAN 00006  19:15:00    23   71 4
   7  G-8 IMG        6 JAN 00006  20:15:00    23   71 4
   8  G-8 IMG        6 JAN 00006  21:15:00    23   71 4
   9  G-8 IMG        6 JAN 00006  22:15:00    23   71 4
  10  G-8 IMG        6 JAN 00006  11:15:00    23   71 4
IMGLIST: done

Next, I tried to see if your point source data files were accessible:

DATALOC ADD RTPTSRC cacimbo.ggy.uga.edu

Group Name                    Server IP Address
--------------------         ----------------------------------------
RTPTSRC                      CACIMBO.GGY.UGA.EDU

<LOCAL-DATA> indicates that data will be accessed from the local data directory.
DATALOC -- done
PTLIST RTPTSRC/PTSRCS.ALL FORM=FILE
PTLIST: No MD files found
PTLIST: Done

This tells me that either no MD files are being created from XCD processes
or that the REDIRECTions in the 'mcidas' account are not valid.


re: setup the 'mcadde' account to have the same HOME directory as 'mcidas'
>OK. This I didn't realize. I also (before rebooting) 
>changed the shell on user mcadde to /bin/false and changed 
>the home directory for mcadde to /unidata/home/mcidas. I 
>also changed my inetd.conf file. I have attached these 
>files.

Very good.

re: testing point source data access
>OK. As before, I had already gone through all of these 
>steps before I sent the question. (Not trying to be 
>smart... just want you to know I'm making an effort.)

I understand completely.  I included the information just in case you
hadn't done the steps.

>I went through the steps again after getting the ADDE server 
>responding. I have attached the LOCDATA.BAT file. I also 
>checked the redirections with a REDIRECT LIST. Everything 
>is set to /data/mcidasd. 
>
>I then tried a SFCPLOT T USA. No dice.

This is not good.  The failure on your side matches my test above.

>I tried a DATALOC ADD RTPTSRC LOCAL-DATA before the SFCPLOT 
>command. Nada.

This must mean that the REDIRECTions are wrong or that MD files are
not being created by the XCD decoders.

>So, I went to the SFCPLT command. Still not finding data. 
>BTW, I see no MD files in my /data/mcidasd directory.

Bingo!

>I will assume this is significant! I have all sorts of files, 
>but no MD files. I have attached a listing of my 
>/data/mcidasd directory.

The other thing I discovered is that I can't access GRID files from
your system.

re: turning on GRID decoding
>I had set up the GRIB decoding before writing. (Again, not 
>trying to be smart...) DECINFO does show the GRIB decoder 
>to be active. 

OK.  I am baffled right now.

>I still wonder about the MD files. I should be getting 
>them.

You should if the following was done:

o added the xcd_run entries to pqact.conf and restarted your LDM

o copied SCHEMA from the ~mcidas/data directory to /data/mcidasd
  (the point source decoders need the information in the schema
  file in order to know how to write the data)

>I still have all of the "old" decoders running in 
>addition to the XCD.

Old decoders?  Do you mean ldm-mcidas decoders?  If so, I need to
remind you that the point source and grid files that used to be
in the Unidata-Wisconsin channel back on July 1 of 1999.  The only
products coming in the Unidata-Wisconsin datastream anymore are
images.

>I have attached my pqact.conf file. 

Great.

>Maybe the problem is there. (A pqactcheck shows it to 
>be OK.)

I am glad that you tested the file contents integrity!

>Some of this stuff is probably out of date since I 
>was gone for over a year. I haven't had a chance to update 
>the pqact.

OK.

>I'm not sure what to try next, but we do seem to be 
>chipping away at problems. 

Yes, we are making progress.  The fact that remote access works to
your ADDE server is good.  The whole ADDE thing is what is so cool
about McIDAS!  To prove this to yourself, try the following from
a McIDAS-X session started as the user 'mcidas':

DATALOC ADD RTPTSRC ADDE.UNIDATA.UCAR.EDU
SFCPLOT T USA
DATALOC ADD RTPTSRC cacimbo.ggy.uga.edu

Pretty cool, huh!?  

Comments on files you attached:

services::

 ...
# UNIDATA stuff 
ldm             388/udp         unidata         # UCAR Unidata LDM 
ldm             388/tcp         unidata         # UCAR Unidata LDM 

mcserv          500/tcp      # McIDAS ADDE port
mccompress      503/tcp      # McIDAS ADDE compression port

Looks good.

inetd.conf::

mcserv  stream  tcp     nowait  mcadde  /unidata/home/mcidas/bin/mcservsh       
mcservsh -H /unidata/home/mcidas
mccompress      stream  tcp     nowait  mcadde  
/unidata/home/mcidas/bin/mcservsh       mcservsh -H /unidata/home/mcidas

This also looks correct.  The routine /unidata/home/mcidas/bin/mcservsh is
a Bourne shell script that will read in information from the file
/unidata/home/mcidas/.mcenv.  '.mcenv', contains McIDAS envirionment
information (e.g. MCDATA, MCPATH, MCGUI, PATH, etc.) that is needed
to run McIDAS applications.  Since I can list out (and display) imagery
from your machine, I figure that '.mcenv' is setup correctly.

LOCDATA.BAT::

This file was zero lenght in your message.

pqact.conf::

My comments are sorrounded by '******************************' below.

#####
#Entries for XCD decoders
#
DDPLUS|IDS      ^.*     PIPE
        xcd_run DDS
HRS     ^.*     PIPE
        xcd_run HRS

******************************
Looks good.  The other things that are needed are (you probably already
did these, but others may read these transactions to see what could
be wrong):

o copy xcd_run to ~ldm/decoders directory

o edit ~ldm/decoders/xcd_run and adjust MCDATA, MCPATH, PATH, LD_LIBRARY_PATH
  to match your McIDAS installation

o make sure that xcd_run has execute permission set
******************************

 ...

####                                                            MCIDAS
# Pattern/actions for the Unidata/Wisconsin (i.e. McIDAS) data stream.
# See the man page documentation for gunrv2, lwfile, lwtmd2, and lwtoa3
# for more info on how to customize these.
#
# GRID files:
#
MCIDAS  ^(GUNRV2 .*)
        PIPE
        -close /unidata/home/mcidas/ldm-mcidas/bin/gunrv2 -d /data/mcidasd -v

# Opaque files:
#
MCIDAS  ^(LWFILE .*)
        PIPE
        -close /unidata/home/mcidas/ldm-mcidas/bin/lwfile -d /data/mcidasd -v

# MD files:
#
MCIDAS  ^(LWTMD2 .*)
        PIPE
        -close /unidata/home/mcidas/ldm-mcidas/bin/lwtmd2 -d /data/mcidasd -v

# AREA files:
#
MCIDAS  ^(LWTOA3 .*)
        PIPE
        -close /unidata/home/mcidas/ldm-mcidas/bin/lwtoa3 -d /data/mcidasd -v

# Profiler files:
#
MCIDAS  ^(LWFILE PROFILER.CDF)
        PIPE
        -close /unidata/home/mcidas/ldm-mcidas/bin/lwfile -d /data/mcidasd -v

MCIDAS  ^(LWFILE) (PROFILER.CDF) .* DIALPROD=(..)
        EXEC
        cdftomd -d /data/mcidasd \3 \2 81

# Fronts for McIDAS
WMO     ^ASUS01 .... ([0-3][0-9])([0-2][0-9])
        FILE    /data/surface/front/ASUS1%j


******************************
You can comment out the entries for GUNRV2, LWFILE, and LWTMD2 above.
Again, the only data in the Unidata-Wisconsin datastream since July 1
is imagery.

If you are going to want FSL wind profiler data decoded into McIDAS
MD files, you need to add the following entries to pqact.conf:

FSL2    ^FSL\.NetCDF\.NOAAnet\.windprofiler\.01hr\.(.*)\..*
        PIPE    -close
        /unidata/home/mcidas/ldm-mcidas/bin/proftomd -v
        -l /usr/local/ldm/logs/ldm-mcidas.log -d /data/mcidasd U2 WPRO 81

FSL2    ^FSL\.NetCDF\.NOAAnet\.windprofiler\.06min\.(.*)\..*
        PIPE
        /unidata/home/mcidas/ldm-mcidas/bin/proftomd -v
        -l /usr/local/ldm/logs/ldm-mcidas.log -d /data/mcidasd U6 WPR6 91

This decoder requires that the McIDAS SCHEMA file be in the ouput data
directory, /data/mcidasd.
******************************


####                                                                 WSI
# Nexrad data
#

******************************
The current release of McIDAS-X has an ADDE server for both NIDS and
WSI NOWrad (tm) data.  It can use the raw files the same way that
GEMPAK/GARP can, so there is no need to turn those products into AREA
files.
******************************

####
# NLDN decode actions
#
#
#       nldn format: YYJJJHHMbMe -- YY year, JJJ julian day, HH hour
#                                   Mb begin minute bin, Me end minute bin

******************************
The format of the NLDN broadcast product header as of January 1, 2000 is:

#       nldn format: CCYYJJJHHMbMe -- CCYY year, JJJ julian day, HH hour

This means that the regular expression for the following NLDN actions
will no longer be valid (they would match 9YJJJHHMbMe.  If you
want to process NLDN data, you should change the regular expressions
from:

^(9[3-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])

to:

^([0-9][0-9][0-9]|[0-9][0-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])

This will match both YYJJJHHMbMe and CCYYJJJHHMbMe formats for the NLDN
data products.

Also, you will probably what to check the pathname for the McIDAS NLDN
decoder as the one in the following lines does not follow what you
have for lwtoa3 above.
******************************

#
# Store 6 min bins with group parts
#
NLDN    ^(9[3-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
        FILE
        -close /data/nldn/\1\2\3\4\5.ingest
#
# Store 6 min bins verbatim
NLDN    (.*)
        FILE
        -close /data/nldn/\1.ingest
#
# Store transmitted, binary form as YYJJHH file
NLDN    ^(9[3-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
        FILE
        -close /data/nldn/\1\2\3.ingest
#
# Decode transmitted binary form as GEMPAK ship file (named YYMMDDHH.gem)
NLDN    ^(9[3-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
        PIPE    /usr/local/ldm/decoders/bin/dcnldn -m 10000 -s minute10
        -d /data/gempak/logs/dcnldn.log
        -p /unidata/home/gempak/gempak5.4/tables/pack/nldn.pack
        /data/gempak/nldn/YYMMDDHH.gem
#
# Decode transmitted, binary form as McIDAS MD file
NLDN    ^(9[3-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
        PIPE
        -close /usr/local/ldm/util/bin/nldn2md -v -d /data/mcidasd 70 NLDN 
DIALPROD=LD \1\2 \3\400 DEV=CNN
#

 ...

******************************
The next two NLDN actions appear to be repeats of ones above:
******************************

# Store transmitted, binary form as YYJJHH file
NLDN    ^(9[3-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
       FILE     -close /data/mcidasd/\1\2\3.ingest

# Decode transmitted, binary form as McIDAS MD file
NLDN    ^(9[3-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
        PIPE
        -close /usr/local/ldm/util/bin/nldn2md -v
        -d /data/mcidasd 70 NLDN DIALPROD=LD \1\2 \3\400 DEV=CCN


******************************
OK, the XCD stuff in pqact.conf looks good.  There some mods needed
as I indicate above, but these would not keep the XCD point source
decoders from working.
******************************


data.mcidasd::

numerous .IDX files

-rw-r--r--   1 ldm      apps         128 Jan  5 18:44 AVM00005.IDX
 ...

numerous AREA files, but no composites.  More on this later.

-rw-rw-rw-   1 ldm      apps      607776 Jan  6 07:34 AREA0060
 ...

.XCD files (these contain ALL of the DDPLUS|IDS data for a single
day):

-rw-rw-r--   1 ldm      apps     80072960 Jan  5 18:59 DD000050.XCD
-rw-r--r--   1 ldm      apps     177427840 Jan  6 18:09 DD000060.XCD

XCD rapid access files:

-rw-rw-r--   1 mcidas   apps      142596 Jan  5 09:39 FOUS14.RAP
-rw-rw-r--   1 mcidas   apps     6617092 Jan  5 09:39 RAOB.RAP
-rw-rw-r--   1 mcidas   apps     26088580 Jan  5 09:39 SAOMETAR.RAP
-rw-rw-r--   1 mcidas   apps     6364676 Jan  5 09:39 SYNOPTIC.RAP
-rw-rw-r--   1 mcidas   apps     4896388 Jan  6 18:09 TERMFCST.RAP
-rw-rw-r--   1 ldm      apps     6346420 Jan  6 18:09 TERMFCST.RAT

HRS spool file:

-rw-rw-r--   1 ldm      apps     16781313 Jan  6 18:09 HRS.SPL

ROUTE.SYS, SYSKEY.TAB, SCHEMA:

-rw-rw-r--   1 mcidas   apps       14080 Jan  6 18:06 ROUTE.SYS
-rw-rw-r--   1 mcidas   apps      432640 Jan  5 09:39 SCHEMA
-rw-r--r--   1 mcidas   apps       24000 Jan  6 18:34 SYSKEY.TAB

The *.IDX and *.XCD files look normal.  The *.RAP files were created
when you ran 'BATCH XCDDEC.BAT'.  Each one of these should have a
corresponding *.RAT file.  The only one I see is the one for terminal
forecasts.  This goes along with you not decoding any point source
data.

The HRS.SPL file shows that model data is being written to the spool,
but the lack of GRID files shows that the GRIB data monitor, DMGRID,
is not decoding any of them into McIDAS data files.

Where do we stand?  It seems that there is a problem running the
point source and grid XCD decoders (duh! :-).  The question is what
is the problem?  To learn more, we both need to look at the contents
of:

o ~mcidas/workdata/XCD_START.LOG
o ~mcidas/.mcenv
o the output from 'DMAP LWPATH.NAM'
  (if this file is not in ~mcidas/workdata, we have a problem)
o the contents of ~mcidas/workdata/LWPATH.NAM
o the output from 'DMAP MD'
o a directory listing of ~mcidas/workdata

Tom