[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20020215: realtime FTPs to motherlode.ucar.edu (cont.)

>From: "Kevin Polston" <address@hidden>
>Organization: NOAA
>Keywords: 200202012111.g11LBCx14907 LDM installation

Kevin,

>Well since your last e-mail here's what's happening.   First the good
>news --  I found out before your last e-mail that I actually was
>receiving some model data!  :-)

Excellent!

>It was being saved in the
>/home/ldm/data/gempak/model directory (which I believe it created).

If the data was written from a FILE action in ~ldm/etc/pqact.conf,
then all directories/subdirectories to a location specified in the
FILE action will be created by the LDM if possible (i.e., if the user
running the LDM has permission to create the directories/subdirectories).
If the action was to run the data through GEMPAK decoders, then
I am out of my area of expertise (I handle McIDAS here at Unidata).

>That was good. For awhile I was getting the RUC model data too but for
>some reason it has stopped coming in.

Stopped coming in, or stopped being decoded/filed?

>It was being saved in a different directory (I believe it was
>/home/data/usr1/nawips/metdat/gempak/grids).  I think I might have done
>something to the RUC lines in the pqact.conf file when I added some
>other lines in but when I checked I got the "syntactically correct"
>statement.....

One thing that 'ldmadmin pqactcheck' used to not be good at finding
was a line that began with a space character.  This is a big no-no
for pqact.conf.  It is easy to find in vi, however:

<as user 'ldm'>
cd ~ldm/etc
vi pqact.conf

/^ 

The / character when vi is not in edit mode means find.  The ^ character
means beginning of line.  The one character you may not see is a space.
The sequence reads:

forward slash caret space

You are telling vi to look for a line that begins with a space.  If
you find such a line, either delete it, or if there is info on the
line that is needed, simply delete the leading space(s).

>so I don't know why I am not seeing the RUC grids.  But
>the good news is it was getting stuff.

>Now for the bad news.  Some of
>this might be due to ignorance on my part like how big the model grids
>are supposed to be.....

The model output comes in grib messages which by themselves are not
that large.  The products that are very large are the VIS images
in the NIMAGE feed (they are approx. 14 MB each) and the VIS and IR
images in the MCIDAS feed (they can be on the order of 1.2 - 1.8 MB).
If we find that receipt of other products is being hampered by these products,
we will need to play some games with requesting them from a different
alias for papagayo.  This was the reason that the request line
for NIMAGE was different that the request line for the other feeds:

request DDPLUS|IDS|HRS ".*" papagayo.unl.edu
request NIMAGE ".*" 129.93.52.150
request NEXRAD "/pN0R" papagayo.unl.edu

These requests are all to papagayo.unl.edu, but two of them,
DDPLUS|IDS|HRS and NEXRAD, are handled by one rpc.ldmd and the
other, NIMAGE, gets handled by another rpc.ldmd.

This brings up a point about your pqact.conf actions included below:

MCIDAS ^pnga2area Q1 (U[^ACXY1]) (.*) (.*)_IMG (6.8)um (.*) (........) (....)
       PIPE    -close
       pnga2area -vl /usr/local/ldm/logs/ldm-mcidas.log
       /usr1/nawips/metdat/images/sat/\3/\5/WV/WV_\6_\7
####
MCIDAS ^pnga2area Q1 (U[^ACXY1]) (.*) (.*)_IMG (0.65)um (.*) (........) (....)
       PIPE    -close
       pnga2area -vl /usr/local/ldm/logs/ldm-mcidas.log
       /usr1/nawips/metdat/images/sat/\3/\5/VIS/VIS_\6_\7

Since you have no request for the MCIDAS feed, you will never get a MCIDAS
product, so your MCIDAS pqact.conf actions will never get executed.

>but when I've moved the gridded data to my
>$METDAT/gempak/grids directory and then tried to display it.......there
>has been lots of missing data. I am aware that I needed to set up my
>datatype.tbl and modres.tbl  and I did do that. But still I am missing
>lots of data.  For instance, I am displaying a map of sfc pressure,
>1000-500mb thickness, wind barbs and dewpoints.  Well, for a few frames
>all the data shows up. Then for a few more frames I might get just the
>surface pressure and wind barbs, then the next frame might just show
>dewpoint and the one after that shows just the thicknesses or more
>often than not.....it is just an empty screen. Well part of the problem
>was I figured I was moving the data too soon before all of the data had
>come in.  But in other instances it seemed as if it was all there (or
>at least the size of the file was no longer growing when I checked) but
>there were still a couple of missing time frames showing up in the data
>selection window and also the problems I mentioned about data not
>displaying at all or partially once it was loaded. So I don't know what
>is going on.

>I am going to try and be more patient and make sure all
>the data has arrived first and I'm also going to check to see how big
>those files are supposed to be.

I would recommend you setting up data ingestion/decoding into a
directory structure that allows you to not have to move the data.
This will need to be done in conjunction with your GEMPAK setup, of
course.  The reason for this is that you can not predict when any
data will come in: it could be held up in broadcast in NOAAPORT, and/or
it could be held up in relay from your feed site's feed site.  If
you setup GEMPAK and the decoders to use the data in place, you will
then not have to worry about moving a data file too soon.

>They should be the same size all the
>time shouldn't they?

If all of the data is available all of the time, yes.

>And a big question I have is how do I get the data
>to go to my proper directory instead of being saved in the directory
>under ldm.

You can tell the LDM and/or GEMPAK decoders to put the data anywhere
you want.  How this is done is based on the particular LDM FILE action
(if the products are simply being filed) or the particular GEMPAK
decoder action.  Chiz's GEMPAK web pages show his recommended setup
for both the GEMPAK/LDM file/decode actions as well as the setup
for GEMPAK itself.  It is easiest to follow these.

>This way I wouldn't have to move the files and they woudl
>get stored properly.

Right, you _definitely_ do not want to move the files around.

>You can see from my pqact.conf file that I've made some more additions.
>However....as of this time I still am not getting any of the 0.5
>reflectivity nexrad data (or at least I can't find it!).

You can always check to see if your feed site is getting the NEXRAD
products by using notifyme as the user 'ldm':

notifyme -vxl- -f NEXRAD -o 3600 -p "N0R" -h papagayo.unl.edu

This requests asks your feed site to list any N0R NEXRAD products it
has received in the last 3600 seconds.

If your feed site is getting the data, and if your feed site has allowed
you to request the data, and if your request of the data is correct, then
you should be fed the data.  You can use notifyme to your own machine
to see if it is coming into the LDM queue:

notifyme -vxl- -f NEXRAD -o 3600

This requests asks your LDM to list any NEXRAD products it has received
in the last 3600 seconds.

>I apparently
>have not done something right in setting up my surface and upper air
>lines either as I can't seem to find that data either.

I am afraid that is definitely a GEMPAK question that I can not field.

>You said the
>satellite data wouldn't come in yet since I don't have anything in
>there to get or decode it.

Right.  You need a routine to unPNG compress the NIMAGE images, and
you need a request for the MCIDAS images.  The MCIDAS request is
an easy one:

request MCIDAS  ".*" papagayo.unl.edu

Assuming that you downloaded the ldm-mcidas decoders (LDM decoders for
the MCIDAS datastream products, that is), then you will be able to
start decoding the images as soon as you make a slight change to
the pnga2area pqact.conf lines:

MCIDAS ^pnga2area Q1 (U[^ACXY1]) (.*) (.*)_IMG (6.8)um (.*) (........) (....)
        PIPE    -close
        pnga2area -vl /usr/local/ldm/logs/ldm-mcidas.log
        -a /usr/local/ldm/etc/SATANNOT
        -b /usr/local/ldm/etc/SATBAND
        /usr1/nawips/metdat/images/sat/\3/\5/WV/WV_\6_\7
####
MCIDAS  ^pnga2area Q1 (U[^ACXY1]) (.*) (.*)_IMG (0.65)um (.*) (........) (....)
        PIPE    -close
        pnga2area -vl /usr/local/ldm/logs/ldm-mcidas.log
        -a /usr/local/ldm/etc/SATANNOT
        -b /usr/local/ldm/etc/SATBAND
        /usr1/nawips/metdat/images/sat/\3/\5/VIS/VIS_\6_\7

A user that has Unidata McIDAS installed under /home/mcidas would not
have to include the '-a' and '-b' flags in the pnga2area actions since
pnga2area was built to look in /home/mcidas/data for these files by
default.  Since you presumably do not have McIDAS loaded on your
workstation, you will have to copy the SATANNOT and SATBAND files from
the ldm-mcidas-7.6.4/etc directory to ~ldm/etc.

One more comment.  The above actions will only decode the water vapor
(6.8 um) and VIS (0.65) um products in the MCIDAS stream.  Presumably
this is what you were after?

>So based on that statement I am assuming the
>two statements in the pqact file starting with McIdas are no good.

They need modification and your ldmd.conf file needs the appropriate
request line.

>I
>must not have the ECMWF, the CMC GEM Model or the NOGAPS set up
>properly either as I have yet to see those model data sets show up.

You have no requests for the CMC GEM model data (which is just
getting setup, by the way), or for the NOGAPS data which is fed
point-to-point from FNMOC.  I think we can worry about those after
everything else is working smoothly.

>My
>final entries were for the SPC products (ie. storm reports, watches,
>etc)....and I haven't seen anything for those either but there is no
>severe weather now so that might be the reason.  Although wouldn't
>something show up for the daily statistics...even if there were no
>reports?  It is a daily file.

Again, I am out of my league here since this is a GEMPAK issue.
Chiz's web pages show appropriate pqact.conf actions for all of
these things, so if GEMPAK is setup correctly, and if the GEMPAK
tables are where the pqact.conf actions claim they are supposed
to be, and if they are readable by the user running the LDM,
then the reports should be correctly filed.

>Finally I seemed to have "fubar'd" a table somewhere. After I ended up
>adding some of these lines into the pqact.conf file I went and edited
>my datatype.tbl file and my modres.tbl file so the model data would be
>able to read it. Well, I must have done something because when I start
>NMAP2 now, no data shows up in the selection window for any of my
>imagery (satellite and radar). I checked the directory itself and the
>data was there. I can access the data using NSAT, NMAP or GARP but not
>NMAP2. So I assumed it was my datatype.tbl file. But looking through it
>I can't find what's wrong!

This is definitely a GEMPAK issue.

>I sent it to Chiz today but so far no reply.

He has been very busy lately.

>When I try to display other data types such as profilers or upper air
>plots they still plot on NMAP2. So I am perplexed and frustrated by not
>being able to get that fixed yet. I'm sure it is just a matter of time
>but I wish I knew what happened.

It is probably something simple.  I will touch base with Chiz on Monday.

>All of this stuff has kept me busy. I have not added the "allow
>ANY...." line into the ldmd.conf yet but I will do that here in a
>couple of hours.

OK.

>Although I don't know if this will let you in or not
>since you can't get in now.

We are assuming that there is a firewall between you and the outside
world.  We are further assuming that firewall is setup to allow
you to get data through port 388 from papagayo, but not to allow other
machines to get access (like *.unidata.ucar.edu machines).

>But I will send you an e-mail when I add
>that in.

Very good.

>Also, once I make changes to my pqact.conf or ldmd.conf files
>do I have to restart LDM?

After making changes to ldmd.conf, you must stop and restart your LDM
NB: after doing an 'ldmadmin stop' you _must_ wait for all LDM processes
to exit before doing an 'ldmadmin start'.  This is important!

After making a change in pqact.conf, all you have to do is:

ldmadmin pqactcheck
ldmadmin pqactHUP
ldmadmin tail

The 'ldmadmin tail' is simply put on the list for you to look at the
end of your ~ldm/logs/ldmd.log file.  It should show you a message
to the effect that pqact is rereading pqact.conf.

>If so is this the proper way to do it so
>those files are invoked.....I usually just go "LDMADMIN STOP" and wait
>for it to say LDM server stopped and then I do "LDMADMIN START" and
>wait for it to say the ldm server is started again.

I would add doing a:

ps -u ldm

between the stop and start.  If you have an rpc.ldmd that is ingesting
a large product, it may take a while to complete.  Do not do a start
until all of the rpc.ldmd processes have gone away.  Again, this is
important!!

>When I initially
>start up LDM I go to the root directory (since it won't let me from
>ldm) and do "Boot_LDM Start" and that seems to start it. (The Boot_LDM
>file coming from what I copied from the unidata ldm webpage).

This doesn't sound right at all.  The LDM should _NEVER_ be run as
'root' (never, never, never...).  If you are unable to start the LDM
with a simple 'ldmadmin start' as the user 'ldm' (the user that
installed the LDM), then something is definitely wrong.  Most likely,
the cause would be that you did not install as the user 'ldm' or
that you did not finish your installation by setting root execute
for rpc.ldmd and hupsyslog.  Here is the snippit from the LDM web
page that describes this:

Make rpc.ldmd and hupsyslog suid root

Change permissions for rpc.ldmd and hupsyslog to be setuid root. This
step needs to be done as root

   % cd ~ldm/{version-directory}/bin
   % chown root rpc.ldmd
   % chown root hupsyslog
   % chmod u+srwx,g+rx,o+rx rpc.ldmd
   % chmod u+srwx,g+rx,o+rx hupsyslog
            
>Looking forward to your next e-mail. It always seems like I make some
>gains after getting an e-mail from you.  :-)

Just to make sure:

1) you did install the LDM as the user 'ldm' didn't you?

2) you did do the setuid root step listed above didn't you?

If the answer to either of these is no, then you need to stop immediately
and get the LDM installation cleaned up.

Tom