[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20020211: realtime FTPs to motherlode.ucar.edu (cont.)

>From: "Kevin Polston" <address@hidden>
>Organization: NOAA
>Keywords: 200202012111.g11LBCx14907 LDM installation

Kevin,

Sorry I couldn't jump on this earlier today, but it has been a busy day...

>I've attached my current ldmd.conf and pqact.conf files as you
>requested. In my pqact.conf file I took a couple of items out and just
>left in that radar line.....just to see if I was getting anything. I'm
>going to list my directory structure on my pc to help you out.

Good move.

Before jumping into pqact.conf items, let's talk about ldmd.conf and
the firewall setup there.

First, it is always a good idea to 'allow' your upstream feed site
to request data from you.  The idea is not that they will try to
feed data from you, but, rather, that they would be able to run
notifyme to your machine and your LDM would respond.  Your ldmd.conf
already has an allow for Unidata machines:

allow   ANY
    ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$))

You need to add one for papagayo:

allow   ANY     papagayo\.unl\.edu$

Next, even though we (Unidata) are allowed in your ldmd.conf, we are
not able to successfully do a notifyme to your machine:

/local/ldm/etc% notifyme -vxl- -o 3600 -h mkc-65-26-24-74.kc.rr.com
Feb 12 00:41:48 notifyme[14768]: Starting Up: mkc-65-26-24-74.kc.rr.com: 
20020211234148.995 TS_ENDT {{ANY,  ".*"}}
Feb 12 00:42:14 notifyme[14768]: NOTIFYME(mkc-65-26-24-74.kc.rr.com): 
h_clnt_create(mkc-65-26-24-74.kc.rr.com): Timed out while creating connection

This seems to be saying that your network is behind a firewall that is
not permitting the notifyme.  This is strange since you are able to
go out through port 388 for the LDM transfers (which come back from
a remote host).  Is it possible that your firewall is setup to deny
port 388 traffic?  I would have thought that papagayo would be allowed
to see your machine, but notifyme from it to mkc-65-26-24-74.kc.rr.com
also fail:

/home/ldm% notifyme -vxl- -o 3600 -h mkc-65-26-24-74.kc.rr.com
Feb 12 00:44:34 notifyme[1289]: Starting Up: mkc-65-26-24-74.kc.rr.com: 
20020211234434.666 TS_ENDT {{ANY,  ".*"}}
Feb 12 00:44:59 notifyme[1289]: NOTIFYME(mkc-65-26-24-74.kc.rr.com): h_clnt_crea

The inability for either of our machines to contact your machine needs to
be addressed; this is the only way we can run some needed troubleshooting
of the IDD connection to you.

Now, on to pqact.conf.

>RADAR DATA - /usr1/nawips/metdat/images/radar/nids
>(For the individual products I have several directories....
>CR for comp. ref; ref1 for 0.5 ref; ref2 for 1.5 ref; vil for vil; srm0
>for 0.5 srm, etc, etc.)

A quick check of your file shows that it is incorrectly formatted.  Certain
white space _must_ be tabs in this file.  Also, your action for NEXRAD
data is incorrect.

Here is what you sent me:

##########################################################################
# Nexrad Data
##########################################################################
WSI     ^NEX/(FTG)/(.*)/([1-2][0-9])([0-9][0-9])([0-1][0-9])([0-3][0-9])([0-2][0
-9])([0-6][0-9])
FILE    -close  $METDAT/images/radar/nids/\1/\2/\2_\3\4\5\6_\7\8

1) the NEXRAD feed that you can get for free in the IDD has an LDM feed
   type of NNEXRAD.  The WSI feed type is for the feed that one can
   contract with WSI Corporation for.

2) the white space between the feed type and the pattern must be a tab.  There
   needs to be a tab before the FILE action and between the FILE and
   -close actions.  Also, there should be a tab between -close and METDATA

3) the LDM pqact.conf file does not understand Unix environment variable
   references, so the $METDAT means nothing.

4) your ldmd.conf file is not requesting NEXRAD data from papagayo.  To get
   all of the N0R products from all NEXRADs (papagayo is getting all of
   the N0R products from all radars), you can add the following request
   to your ldmd.conf:

request NEXRAD "/pN0R" papagaoy.unl.edu

   If you want to get all of the NEXRAD products that papagaoy gets,
   you can change your ldmd.conf request to:

request NEXRAD ".*" papagayo.unl.edu

   In this case you should be aware that papagayo only gets all NEXRAD
   products for a limited set of radars.  Here is papagayo's requests
   line for its feed of NEXRAD products:

request NEXRAD  "/p...(OAX|UEX|LNX|FSD|TWX|GLD|CYS|MRX)" stokes.metr.ou.edu
request NEXRAD  "/pN0R" stokes.metr.ou.edu

   These say: get all products from OAX, UEX, LNX, FSD, TWX, GLD, CYS, and
   MRX and also get all N0R products.
   
   If your machine has enough disk storage, and if your internet
   connection can handle the volume, then you might want to replicate these
   requests from papagayo:

request NEXRAD  "/p...(OAX|UEX|LNX|FSD|TWX|GLD|CYS|MRX)" papagayo.unl.edu
request NEXRAD  "/pN0R" papagayo.unl.edu



Next, the action your pqact.conf file should have to file all NNEXRAD
products in the /usr1/nawips/metdat/images/radar/nids directory would
look something like:

NNEXRAD ^SDUS[2357]. .... ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
        FILE    -close  
/usr1/nawips/metdat/images/radar/nids/\5/\4/\4_(\1:yyyy)(\1:mm)\1_\2\3

That is:

NNEXRAD<tab>^SDUS...
<tab>FILE<tab>-close<tab>/usr/...

where <tab> indicates a tab character.
   
You can check the integrity of actions in your pqact.conf file using
the ldmadmin script:

<login as 'ldm'>
cd etc
ldmadmin pqactcheck

Running this on your current file should result in ldmadmin telling
you that the file has problems.

>SATELLITE DATA - /usr1/nawips/metdat/images/sat/GOES-8/1km/VIS/
>or                                                    /4km/IR/
>etc, etc.....
>
>SURFACE DATA - /usr1/nawips/metdat/gempak/surface
>UPPER AIR DATA - /usr1/nawips/metdat/gempak/upperair
>PROFILER DATA - /usr1/nawips/metdat/gempak/upperair/
>MODEL DATA - /usr1/nawips/metdat/gempak/grids

Since you are a GEMPAK user, I would follow Chiz's examples for filing
data and running GEMPAK decoders.  He elucidates these in the GEMPAK
web pages.

>I followed your examples in your e-mail that you sent me. I noticed in
>my ~ldm/data directory there is a file called ldm.pq that is quite big.

Yes.  The LDM request data from one or more upstream feed sites, and
the products it gets are put into the product queue, ldm.pq.  The
queue file size is set in the script ldmadmin.  The queue does not
grow in size once it has been created.

Products are read out of the queue and acted on by pqact (Product Queue
ACTion).  pqact uses entries in ~ldm/etc/pqact.conf to determine what
to do.

>Am I correct in assuming that file is where data is being written to?

Written to in the "staged" sense.  The queue is a like a circular
buffer.  Products are written into it; acted on by pqact.conf; and aged
out.

>Then I need the pqact.conf file to process that data?

Yes.

>I must have had
>something working briefly because the first time I got the LDM going all
>of a sudden a bunch of directories were created in my ~ldm/data
>directory (directories such as RA, RF, RL,RO,RS ....and US.

This was due to your pqact.conf file having the default action
of filing everything:

WMO     ^([^H][A-Z])([A-Z][A-Z])([0-9][0-9]) (....) ([0-3][0-9])([0-2][0-9])
        FILE    data/\2/\4/\6/\1\3.wmo

>I assumed
>those were country directories and when I switched to the US directory
>then I had a bunch of directories such as KEAX, KMSP, KTOP, etc, etc.

No, they are not country directories.  The are directories whose names
relate to the product header.

>In
>those directories was another directory named 02. Then some data files
>were saved in them...although they were not the same data files in all
>directories.  I have been unable since then to duplicate what I did to
>get the data to get saved nor am I sure how to "pipe it" to the proper
>directory (I guess this is where pqact.conf comes into play). 

Your inability to recreate the first success is undoubtedly due to the
incorrect format of the pqact.conf action(s).

>From one of the Unidata web pages which tells you how to configure the
>LDM there was a section called  "Start LDM at boot time".  I copied that
>script and saved it as a file called "boot_ldm" which I use to start
>LDM. I initially did not have LDMBIN, DECODEBIN, and UTILBIN defined
>although I did do that. Also, DECODEBIN is defined as $LDMHOME/decoders.
>There is nothing in my decoders directory.  My decoders are in
>/usr1/nawips/exe/linux  (dcmetr, dcuair, etc). This has me confused  -
>do I need to point DECODEBIN to that directory or is the other directory
>needed for some reason. 

The idea here is that the PATH for the user 'ldm' needs to include the
directory containing decoders.  The typical setup of Unidata users is
to have either/both of ~ldm/decoders and ~ldm/util.  Each/both of these
directories is then include in the PATH definition in the user 'ldm's
.cshrc file.

>Thanks for helping me out with this Tom. I'm sure this is a piece of
>cake for you but I am learning as we go along here. 

You will get the hang of this pretty quickly.  You should follow the
examples on the GEMPAK web pages and modify directories where
appropriate/needed.

>I've got one other question for you and this might be more suited for
>Chiz but I think you could probably help me to.  I want to convert into
>GEMPAK format archived upper air data from historical severe weather
>event days. I've attached a file from the March 13, 1990 tornado
>outbreak (the Hesston KS day)  of the upper air data from 12Z that
>morning. I want to be able to plot upper air maps and also run an
>objective analysis to create height contours, temperature and dewpoint
>contours, and also lifted index and/or CAPE fields, CIN, and shear
>related parameters (such as SRH, BRN, and 0-6km shear).  How exactly do
>I go about doing that?

Chiz really is the guy to answer this one.  I am the McIDAS support
person here.

>I still get confused over how to do some things
>and this is one of them.  When I save the file I want to give it a name
>that shows its date (ie; 900313_12Z) but then how would I get that to
>display in NMAP (would I have to give it a name with the current date?
>If I did that would write over the current upper air data). If you can
>help me I would once again appreciate it. 

I will ask Chiz to take a look at this when he gets a chance.

>Well, I'm sure there will be more but for now thats what I've got. I
>will be looking forward to your next e-mail.  :-) Thanks again for all
>your help. 

As a wrapup, please do a couple of things for me:

1) look into any firewall setup that would prevent us from doing
   notifymes to your machine

2) change your pqact.conf action and see if you don't start filing
   NEXRAD products.

Later...

Tom