[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20020314: IDD data late (cont.)

>From: "Kevin Polston" <address@hidden>
>Organization: NOAA
>Keywords: 200203112125.g2BLPSa16982 IDD

Kevin,

I've been in meetings for two days.  I am now catching up on the backlog
of Support email.

>I stand corrected about the national VIL product.  When I heard you say
>you and Chiz were discussing it I assumed it meant you were working on
>it. When you made the comment about "what do you mean by very soon" I
>read between the lines and thought you meant it was about ready to go.
>So I stand corrected and will try not to get to over-eager again in the
>future.   :-)

Sounds good.  If you are subscribed to any of the following
Unidata-maintained email lists:

gembud
ldm-users
mcidas-x

you will get notification if/when products like the composite VIL are
available.  All you have to do is stay tuned...

>I have attached a copy of my ldmd.log file from this morning. I noticed
>there were a couple of errors in there and thought you might want to
>see them (as opposed to me trying to write it out incompletely).

It seems that readpng is occasionally having problems uncompressing
the NIMAGE products.  Right off, I don't see what could be wrong.

>However....I have to say that since my last e-mail to you the other day
>the system has been working great and the data have been coming in in a
>very timely manner for the most part. Everything seems relatively
>current. By that I mean I suspect there is a usual lag time for some of
>the data but it is within an acceptable time limit. Of course,

OK.

I do
>have a question for you.  :-)   I've noticed on the GOES-8 imagery,
>starting around 06Z, there is a period of time when the images only
>come in every 30 minutes.  This seems to correspond to the times when
>the eastern portion of the image is "blacked out". I am pretty sure
>this is the satellite eclipse period and I am guessing that's why the
>data is only there at 30 minute intervals instead of every 15 minutes.
>But I would just like to get some clarification to see if I am right.

Yes, we are in the middle of the GOES-East (GOES-8) solar eclipse
period, and you are seeing the effects of it.  By the way,
announcements of relating to situations like this are available in the
textual streams (as admin messages) you are receiving in the IDD, and
you can also subscribe to the nws-changes email list we maintain to get
the same information delivered to you in the form of an email.

>My next project here in the continuing ldm journey is to revisit the
>filing of the data so I can get it into the directory structure I have
>set up on my system. I believe if you were to look at my last
>pqact.conf file  I sent you (which should be a couple of e-mails
>ago).....you will notice I have two entries for the NOAAPORT images.
>One of them is commented out. That was the suggestion you sent me to
>configure that entry to get the data to store into my directory
>structure.  Well, I tried it but for some reason it wasn't working
>quite smoothly. Here is what happened. If I clicked on the
>GOES-8/EAST-CONUS/ 4km/IR button (from NMAP) I would get the Puerto
>Rico sector showing up (this was when I was ingesting all images).
>However...if I left it running after I got it open then the proper
>images would start showing up.  But I was not able to go back to the
>menu and choose another sector or type of image because I didn't have
>that option.

We need to start separating GEMPAK/NMAP issues from LDM/IDD ones.
Since I am no expert in GEMPAK, I have no way of knowing if some
of your probles are caused by its setup or not.

>So I reverted back to the old pqact entry and left that in
>there commented out to try again later. Well it is "later" and I will
>give it a try again but if you  had any other suggestions before I
>start I could probably use them.  As it stands now I am getting the
>data starting off in the " _20020314" (or whatever day it is) directory
>and then going from there.  I would like to try and fix that so I don't
>have to mess around with that anymore.

I have given you several examples of how to look at the product
headers of the images that are coming into your system (using notifyme).
From my perspective, what you need to do is:

1) write down exactly what you want the directory structure to look
   like

2) examine the product headers of the image products and see if pieces
   of the header match the various directories in the structure you
   want

3) if there is a match, use the pqact.conf regular expression matching
   to tell readpng to uncompress the images in the desired directory

>Now that I am getting data successfully in a timely manner I would like
>to ask your opinion on one other thing. Being that meteorologists are a
>greedy type and want more data :-)  ......  I am wondering about how
>much more data I could grab and still avoid the "bottleneck"?

There is no absolute answer for this since it depends on situations
that change, the most important of which is network slowness.

>If I
>left my radar at the same level but went back to grabbing all the
>satellite data do you think I could get away with that.

I have no way of telling since I can't do notifymes to your machine
at various times of the day to get an idea of how late the various
products are.

>Or if I left
>the satellite data where it is now but upped some of my radar data how
>much do you think I could go?

Same comment as above.

>I am just asking to get an approximation
>of where I should start since it seems like I will have to play around
>with different combinations and tweak things to see what I can do. Of
>course, it is not bad where it is right now either.

All you can do is experiment.

>Thank you for your explanation on the multiple rpc.ldmd files. It did
>help me understand better and I think I see now why it wouldn't cause a
>slowdown.

Super!  That whole situation can be a little murky when first encountered.

>Well, thats all for now......on a scale of 1-10 things are
>running above a 9 so thats good.  Oh....before I forget.....the model
>data seems to be ingesting smoothly 95% of the time too. It looks like
>it is timely and all the "fields" seems to be there.

The thing with the model data that is in the IDD (coming straight from
NOAAPORT) is that the same gridded fields are in there in some cases
in several different projections.  Unless one really needs all of
the different projections (like, for instance, if you are doing
modeling), it makes good sense to cut down on the products being
requested, and, so, save bandwidth and processing cycles on your
own machine.

>One problem I did
>have this morning was trying to display the MRF 500 mb heights and
>vorticity. I had some missing images and fields but not at all times
>but just some times. All other models looked ok.

>Quick question....is
>the NOGAPS data being sent out on the data stream or not.

The NOGAPS data is available in a point-to-point LDM feed directly
from FNMOC.  You have to arrange to get this feed. The contact person
at FNMOC to set up the IDD feed is:

address@hidden

>I believe I
>have that set up and I don't think I have it commented out. Same for
>the Canadian model (CMC).

The GEM model output (Canadian model) is not yet flowing continuously.
Also, I _think_ (not positive) that it will also be a point-to-point
LDM feed from its source.  If I had a contact for this, I would send
it along.

>Talk to you later Tom.....have a good day.

Later...

Tom

>From address@hidden Sun Mar 17 12:44:21 2002

Tom,

Hi.  Well, here is the deal. I changed the pqact.conf entry for the
NOAAPORT images to the way you suggested (but only for one line). But I
was still having problems. Such as the wrong map projection on the
image, not getting the wrong image on a certain scale, etc.  So I had
another e-mail all set to go to complain about my problems again when I
looked back through your e-mails and found the one you sent which
detailed what the headers mean. Because I didn't understand what they
meant. Well, all of a sudden the "lightbulb" popped on inside my head
and I set up several entries for the different image types. That seemed
to do the trick. :-)  The only thing that is different now is when the
eclipse period comes up it won't display the goes-8 imagery whereas
before I would still get it but with a portion of the image blacked out.
Otherwise....ldm is running very well.  :-)  

I will start "experimenting" with radars and satellite data over the
next few days to see where the "saturation point" is. But since things
are running so well I don't think I will make to much in the way of
extensive changes but I know if I do I can go back to where I'm at now.
Thanks for your help. Sorry I didn't understand sooner. But it is nice
to finally be able to understand things more clearly.
  

KP