[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[ldmMcidas #KQH-987099]: Make error: ldm-mcidas on Mac OS X 10.5.1

Hi Dave,

re:
> Whew! That looked like fun, I think. I'm just glad I wasn't the one
> having it (the fun), particularly since I wouldn't have been able to
> figure it out.

It really wasn't that hard getting things to build on virga.  My email was
long because I wanted to document "for the files" what I ran into so that
I would have something to refer to when I get back to working on the
ldm-mcidas package.

re: crontab to rotate ldm-mcidas.log files

> OK, I'll take care of this. (I've encountered some indications that
> cron jobs, or at least some cron jobs, run successfully but deposit an
> error-like message in the system.log file, and one way around this is
> to construct XML files for the Mac OS X 10.5 launchd program to read
> and run in lieu of cron jobs. I'll look into that, too.)

Interesting.  I have been edging towards buying a Mac for a home system,
so comments like the XML files for launchd are intriguing.

re: adding a redundant feed of UNIWISC data to idd.unidata.ucar.edu

> That's odd. It had to have been working before you added the second
> request because there aren't any gaps in MCIDAS file sequences saved
> on virga.

I didn't think there was a problem; I guess I got impatient.  I had a
notifyme running on my home system looking at idd.unidata.ucar.edu, and
another one running on virga looking at its own LDM queue.  After I saw
indication that idd.unidata.ucar.edu had received 5 UNIWISC images, while
virga had not, I decided to add the redundant feed to idd.unidata.ucar.edu.
It could well have been the case that if I had waited for a couple of more
minutes, the data would have shown up on virga.

> The other odd thing is that the file sizes are the same
> whether the images were saved with or without having been decoded by
> pnga2area. I'll play around with this some more.

This brings up the point of confusion that I had.  The actions I saw for
processing UNIWISC data in your pqact.conf file specifically ran 
/usr/local/unidata/decoders/bin/pnga2area.  Since there was no pnga2area
in /usr/local/unidata/decoders/bin when I first logged on, all of these
actions should have failed.  I was very surprised when I saw numerous,
current images in subdirectories of /data/mcidas.  My theory as to
how these images were getting written to those directories was that
some other machine was also running an LDM and decoding the UNIWISC
images to the same set of directories.  This is why I made the comment
that if this were true, it would have to be attended to as the two
different processes writing the same files might result in corrupted
images.

re: functional i386 version of pnga2area on virga

> Awesome--thanks! Though you note below that it turns out not to be
> necessary for a couple of reasons. :-(    I inadvertently discovered
> one of the reasons yesterday when the LDM was saving images even
> without a working version of pnga2area and WXP (which generates images
> for our Web server) seemed to be displaying them just fine. (Not sure
> why they were getting saved at all, given that they are supposedly
> getting PIPED through the decoder first.)

Yes, exactly.  How were the images being decoded when the decoder specified
in the pqact.conf UNIWISC actions was /usr/local/unidata/decoders/bin/pnga2area
and that decoder did not yet exist!?

> I didn't know that the IDV could read the compressed files. I asked
> Don and Jeff for this ability last summer but it didn't sound like it
> was going to get a high priority.

Please be aware that I am not _sure_ that the latest IDV can read the compressed
images.  I can say, however, that after I mentioned that I modified McIDAS to
read the PNG-compressed AREA images directly, both Don and Chiz talked to me
about what I did.  Chiz implemented direct reads in GEMPAK, and I _thought_,
but was not sure, that Don implemented the same in the IDV.

> Maybe it was there all along and I
> didn't express what I was after very well. I do notice that the IDV
> will read both uncompressed and presumably compressed files (though
> they're the same size on my machine), but no date/time is included in
> the display list at the bottom of the display when the uncompressed
> image is displayed, whereas it does appear when the compressed image
> is displayed.

This is _VERY_ weird especially since the header portion of the AREA files
do not get compressed.  The header portion is where the various bits of
information about the image live (e.g., date, time, size, band, etc.).

re: Is another machine decoding the same images into the same directory
hierarchy?

> No. Virga's predecessor (a PowerMac G5) saved all the other images
> right up to the point where I turned it off and turned on the new
> machine yesterday evening.

OK.  And you renamed the new machine virga... true?  It must be true
since 'uname -a' on virga shows that it is an Intel-based machine (i386):

Darwin virga.sfsu.edu 9.1.1 Darwin Kernel Version 9.1.1: Fri Dec 14 19:00:14 
PST 2007; root:xnu-1228.1.30~1/RELEASE_I386 i386

re: the PPC version of pnga2area under /Volumes/virgabackup/...

> Thats from a backup of the PoweMac G5 version of virga. I switched the
> external hard drive used for backups from the old virga to the new
> one, so the old virga's files (notably data files saved by the LDM,
> scripts, etc.) would be available for configuring the new one.

OK, the light goes on :-)

re: Interestingly, your Intel-based MacOS can run PPC executables:

> Interesting! I knew that many applications designed to run on PPC Macs
> will run on Intel Macs, but only through a layer of software called
> Rosetta, which slows the applications down, typically to the point
> where the superior performance of the Intel chips is canceled by the
> inefficiency of having to run through that extra layer of software. It
> never occurred to me even to try running the PPC version of pnga2area
> on the new machine, though.

Me either until I tried to figure out why there was no big gap in
images processed on virga.  Thats when I ran 'locate pnga2area' and
found the PPC version over under /Volumes/virgabackup.  For grins, I
tried executing it fully expecting the invocation to fail.  I was pleasantly
surprised when it ran...


re: building pnga2area on virga was apparently not needed at all!

> Except perhaps for a modest speed penalty.

And me finding the warts in the ldm-mcidas package for MacOS-X systems.

re: the building exercise was valuable

> Besides, what better way to spend a Saturday morning? :-)

LOL.  Yea, I routinely do some catching up on Saturdays.  Today, I had planned
on doing some support and then upgrading my Linux machine from Fedora Core 6
to Fedora 8.  The only problem was that the motherboard BIOS needs to be updated
to recognize the DVD drive as a bootable device.  Upgrading BIOS is not one of
my favorite things since I torched one motherboard by doing something stupid.

re: changing pnga2area actions to FILE unless other applications need 
uncompressed
images

> It looks like WXP will do the same, so I will indeed give this a try.

I would be surprised if Dan added direct support of the PNG-compressed images.
Then again, Dan is a very clever guy...

> I'm thrilled to see the IDV displaying the MCIDAS images, too--the
> last time I tried that (a year or two ago) it failed, but  being able
> to display MCIDAS images more than a couple of days old (stored on our
> local machine) opens up potential new uses for the IDV for us.

It would be even better if you installed McIDAS and ran an ADDE server for
your data.  Remote access to data via THREDDS and ADDE is very cool and
useful.

> Great work, as usual--it's inspiring to see you maintaining a high
> level of professionalism year after year!

Thanks for the kind words!

By the way, immediately after seeing your reply, I SSHed to virga from my home
machine to reinvestigate the decoding mystery.  After only a couple of minutes, 
my
SSH session froze.  Pings to virga from  laraine.unidata.ucar.edu failed as did
traceroutes (pings routinely fail when folks turn off ping responses in
firewalls, so this did not suprise me).  After a few more minutes, my SSH 
session
exited, and renewed attempts to reconnect resulted in:

[mcidas@zeldagillroy Desktop]$ ssh address@hidden
ssh: connect to host virga.sfsu.edu port 22: Connection refused

After a few more minutes, my ability to SSH to virga returned.  I see that the
machine was rebooted, and that admin is logged in from seward.sfsu.edu.

OK, a new comment that you may want to think about:

- your UNIWISC actions use the system clock to name the output files.  Here
  is your action for GOES-West IR (10.7 um) images:

UNIWISC ^pnga2area Q1 U5 (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close 
        /usr/local/unidata/decoders/bin/pnga2area -vl logs/ldm-mcidas.log
        -a /usr/local/unidata/ldm/etc/SATANNOT 
        -b /usr/local/unidata/ldm/etc/SATBAND
        /data/mcidas/gwir/%%y%%m%%d%%H00_sir.mca

  This prevents your system from saving all of the images you receive (the
  majority of the UNIWISC images (the Vis and all IRs) are sent twice per hour).
  Your action causes the last image of a certain type received in an hour to
  overwrite the first one received in that same hour.  It also could give one
  the wrong idea about the time of an image if one thinks that the output file
  name represents the time of the image.

  If you want to process and save all of the images you are receiving, I
  recommend you change your UNIWISC actions.  A representative header
  for a GOES-West IR image is:

  pnga2area Q3 U5 131 GOES-11_IMG 10.7um 4km 20080210 0130

  Here is a reworked example for the GOES-West IR image action listed above:

UNIWISC ^pnga2area Q1 U5 (.*) (.*) (.*) (.*) (..) (......) (....)
        PIPE    -close 
        /usr/local/unidata/decoders/bin/pnga2area -vl logs/ldm-mcidas.log
        -a /usr/local/unidata/ldm/etc/SATANNOT 
        -b /usr/local/unidata/ldm/etc/SATBAND
        /data/mcidas/gwir/\6\7_sir.mca

  Changes are:

  - change in pattern match regular expression
  - change to use matched patterns for date ('\6') and time ('\7')

  If you don't want to change your pqact.conf actions, you may want to
  consider only requesting the images you want from your upstream site.
  Currently, you are receiving all of the UNIWISC images and virtually
  throwing away half of them (by overwriting the output file) even
  though you are processing all of them.


Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: KQH-987099
Department: Support ldm-mcidas
Priority: Normal
Status: Closed