HI Bas, re: > Mr. Valentijn Venus has asked me to adjust the McIDAS files in order to > decompress PNG-compressed AREA files on-the-fly. At the end of this > e-mail, I have included an e-mail with previous communication between > Unidata support and Mr. Valentijn Venus. OK, thanks for including the previous email. > After spending quite some time learning Fortran (I have a C/C++ > background, and it was the first time that I really needed to analyse > fortran code), Luckily, the code that needs to be enhanced is in C, not Fortran :-) > and understanding the necessary parts of the McIDAS > source code, I have come up with the following solution: > > 1. Make some adjustments in file mcread.c so that it decompresses > PNG-compressed AREA files (I still have to think about how to let mcread > detect whether a file is a compressed PNG or not) The PNG-compressed portion of the images in question are recognizable by a "magic word" located at the beginning of the compressed portion of the image. Also, the AREA file header has been modified to put the comment blocks after the header components instead of at the end of the image. Either/both of these clues can be used to alert the reader that there is a need to uncompress the image. > 2. If needed, make adjustments to the corresponding Fortran files that > call the mcread-functions (like RDTRKA.for, REDARA.for and AGETSERV.fp) > - however, I think if mcread is intelligent enough, no adjustments are > needed in other files. I agree. There should be no need to modify the Fortran routines. The C read routine can be made intelligent enough to recognize the need to uncompress the PNG-compressed data portion of the AREA file. The only reason I did not do this was sites were already used to including uncompression actions in their ~ldm/etc/pqact.conf processing file. Please don't misinterpret this comment: I think adding uncompression on the fly would be a fantastic enhancement to the ADDE serving of the data! > 3. mcread will call a still-to-be-implemented PNG-decompress function, > that looks similar to the main function in pnga2area.c. My comment: instead of making the function look similar to the main function in pnga2area.c, why not make it the same function? In order to add the ability to decompress the PNG portion, we will need to add the PNG library to the standard McIDAS build, and it will be very easy to simply include the routines we already wrote into the McIDAS library. > Most important > difference is that the output is a buffer (not a file). I did not decide > yet where I will implement this function (in mcread.c, in pnga2area.c or > in a new file). OK. I included zlib uncompression in the ADDE server service routine for the GINI ADDE server. > As you may understand, I have chosen for the easiest possible solution > (with the knowledge that I currently have). However, I am a bit > concerned about the performance of my proposed solution. It would work > well if AREA files are read once and then buffered in memory by McIDAS. > What I am worried about is that currently AREA files are repeatedly > opened and closed on disk, while McIDAS reads them line-by-line. In this > case, with the PNG-decompression included, when McIDAS requests a line, > a file will be opened, read entirely, and then decompressed into a large > memory buffer, of which only one line is used. > > My questions: > Could you please tell me if McIDAS avoids repeatedly reading files from > disk, by buffering files that are recently read? If not, I would have to > reconsider my solution. It does _not_. Each time a file is read, it will need to be uncompressed. > Could you also inform me if you have a better solution, or do you have > any suggestions to improve upon the solution mentioned in this e-mail? One implementation of the GINI ADDE server called the uncompress procedure to uncompress only the lines requested by the client request. This mandated that I be able to calculate how much of the compressed portion of the image to skip over before beginning uncompression. The problem I ran into was there was no foolproof way (or at least I did not find one) to jump to the appropriate portion of the compressed image to start. After working on this for over a week, I decided to implement a different scheme where I would start decompressing at the beginning of the compressed portion and continue only as long as needed. This approach is efficient for client requests for portions of the image near the beginning, and very inefficient for client requests for portions of the image near the end. I really worried about the overhead of doing this until I exercised the server code thoroughly. After testing I became convinced that overhead -- while significant -- was not a "show stopper" (i.e., it was something that I could live with). So, I would recommend the same approach: only uncompress enough of the image to satisfy the client's request for data. Uncompressing the entire image at the beginning would most likely be OK for small images, but it would surely be unacceptable for large ones (like full resolution, full disk visible images from GOES or METEOSAT which can be upwards of 300 MB). If, however, we can figure out exactly where to jump into the PNG-compressed portion of the image before starting to uncompress, we would be able to only uncompress the portion of the image that was requested. Even this approach has its downside: if the user wants to display the entire image (or a "blowdown" of the entire image), the entire image will have to be uncompressed. This doesn not to be done before hand, however, it can be done incrementally so that the size of the uncompress buffer can be kept to a minimum. > Thanks in advance, Question: are you actively working on this now, or are you in a thinking phase? You may or may not know that I am in the process of putting together the Unidata McIDAS-X v2006 distribution. It would be wonderful if an enhancement to the AREA ADDE server could be incorporated in v2006's initial release, but I do not think that this will be possible as I need to have something for sites to download by the end of July/beginning of August, and I have not yet started on the huge task of converting SSEC McIDAS documentation to use in our website. It seems to me, therefore, that this code modification (and the code added by you (or Tyn?) to deal with the wavelet-compressed METEOSAT images would be best added in the first addendum to the code. Aside: I am also waiting on some code from CPTEC (a division of Brazil's INPE) that might be used in an ADDE server for imagery data in TeraScan (tm) TDF format. Review of this code (when I get it), turning it into a server, and testing will surely take at least a couple of weeks, so I do not expect it to be included in the initial v2006 release. We will surely continue this discussion in the near future. Please don't hesitate to contact me (address@hidden) to discuss strategies and coding. > Bas. > -- > Ir. V. (Bas) Retsios > Software Developer > IT Department, Sector Remote Sensing & GIS > International Institute for Geo-information Science and Earth Observation > (ITC) > P.O. Box 6, 7500 AA Enschede, The Netherlands > Phone +31 (0)53 4874 573, telefax +31 (0)53 4874 335 > E-mail address@hidden, Internet http://www.itc.nl Cheers, Tom **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: YEF-379651 Department: Support McIDAS Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.