[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20020220: realtime FTPs to motherlode.ucar.edu (cont.)

>From: "Kevin Polston" <address@hidden>
>Organization: NOAA
>Keywords: 200202012111.g11LBCx14907 LDM installation

Kevin,

>This is going to be more of a good news - less bad news e-mail.
>First....I got most everything to work right!

Alright! :-)

>I don't understand why
>because I thought I had already done this but I "re-changed
>permsissions" again and making sure all the settings were right I got
>the data to come in and start storing where it was supposed to be!
>:-)

On occasion, I have seen instances where a site sets up a user account
in one group and then changes the group the user belongs to after
setting file/directory permissions.  The effect of this is that the
group settings for the files/directories don't match the current
information for the user; this can cause permission problems.

>So that made me a happy camper. I had the surface observations
>coming in along with the profiler data and the text messages. So that
>was good. I noticed earlier today that my surface obs were not coming
>in and I believe I figured out why. When I tried to get my upper air
>pqact line set I forgot to take the comment out of a line that was
>supposed to be...therefore my surface and upper air lines were running
>together. But I believe I have that fixed now.

Sounds like you are getting the hang of pqact.conf entries.

>So that is all good. Now
>for the "bad", relatively speaking.  I still am running behind on
>receiving radar data. It had improved this morning to being "only"
>about 30-35 minutes behind the actual time when it had been running
>anywhere from 45-60 minutes behind. I also noticed a similar problem
>with my surface obs....it seemed they wouldn't show up until more than
>half past the hour. But they did all come in...it just didn't seem as
>timely as I thought and perhaps that is related to the delay from the
>other end. I sure hope that improves.

I havn't had a chance to look into why the radar was/is running so
late, but my test the other day showed that papagayo was getting the
data late from its upstream host, and the upstream hosts feed was also
late.  So, you were seeing the result of some sort of a slow link
problem two levels above yourself.  Delivery of radar data an hour
late is certainly not what we strive for, nor is it the norm.

>I turned off most of the model data ingestion. I don't know why but
>once again it seems that despite having large file sizes there was very
>little there in the way of data to look at. I wonder if it has to do
>with max number of grids allowed. That is the only thing I can think of
>off the top of my head right now. But it was very inconsistent so I
>turned on my cronmaster and edited my pqact.conf file to just get a few
>models I normally don't get. I suppose this might be ok since that way
>I can cut down some on the model data and maybe concentrate more on the
>imagery.

We need to figure out your model ingestion problem.  papagayo is not
getting the model data late, so there is something else afoot.

>Speaking of which.....what do I need to do to get the NOAAPORT
>gini imagery going? I feel good now that I got this other stuff going
>and am ready to give it a try. (I hope I am ready to give it a try.
>:-)

You need to grab the executable 'readpng' from the pub/binary/linux_2.2.5-i686
directory of anonymous FTP on our FTP server, ftp.unidata.ucar.edu.

You then need to setup uncompressing of the PNG compressed imagery with
actions that look like:

#
# NOAAPORT GINI Images
NIMAGE  ^sat/ch[0-9]/.*/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        PIPE    -close
        util/readpng -n -l logs/png.log 
data/pub/raw/gini/gini_\2\3\4\5/\8/\9/\1/\1_\2\3\4\5_\6\7

This assumes that you:

1) FTP readpng in binary mode
2) put it in the directory ~ldm/util
3) ~ldm/util is in the PATH for the user running the LDM
4) you want to file the imagery under the ~ldm/data/pub/raw/gini directory

You will need to make modifications to suite your setup where necessary.

>So to sum things up.....I got most things working which is good. The
>bad is the model data and the products being a little bit behind time.

Let's see how the decoding of the GINI images goes.

>I will watch and see how it goes over the next couple of days to see
>how well things are going....hopefully adding some satellite products
>in as well. I have a class I have to teach next week so I am going to
>have to concentrate on it by the end of this week and early-mid next
>week.

Please give readpng a shot.  I will be out tomorrow, but I will read
email in the evening (after getting back from skiing :-).

>Thanks Tom for helping me out.  I feel I have a little bit better
>understanding of how things work now and that has given me a little
>more confidence.

No worries.  It seems that we are close to getting things working
smoothly.  At that point, I will want you to cutoff the realtime FTPs
to motherlode.

>Hopefully down the road we can talk about getting
>McIDAS set up as I will probably look into that in the next couple of
>months.

Well, from my point of view McIDAS is easy to get working.  But, then
again, that is because I am intimately familiar with the package and
know all of its pitfalls.

Tom