Re: I solved the Nexrad mystery! (was: Can rad handle new stuff...) (fwd)

NOTE: The decoders mailing list is no longer active. The list archives are made available for historical reasons.



==============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
rkambic@xxxxxxxxxxxxxxxx                   WWW: http://www.unidata.ucar.edu/
==============================================================================

---------- Forwarded message ----------
Date: Tue, 07 Nov 2000 23:08:01 -0600
From: Pete Pokrandt <poker@xxxxxxxxxxxxxxxxxxxxxx>
To: DeVietor@xxxxxxx
    sebenste@xxxxxxxxxxxxxxxxxxxxx, mwdross@xxxxxxxxxxxxxxx,
    kwthomas@xxxxxxxxxxxxxxxxxxxxxxxx, ldm-users@xxxxxxxxxxxxxxxx,
    support@xxxxxxxxxxxxxxxx, wxp5@xxxxxxxxxxxxxxxx
Subject: Re: I solved the Nexrad mystery! (was: Can rad handle new stuff...) 

Dan and all,

Well, It's pretty ugly still, I wrote some comments to what's
going on, but here's what I used. This script *does* strip
off the uncompressed header to begin with, but does *not* strip off the
header that shows up at the beginning of the uncompressed data after the
first call to inflate. I did that by hand, but another call to substr prior to the first print to STDOUT would do it. 

Pete

#------- Cut Here --------------
#!/usr/bin/perl
use Compress::Zlib ;
$input = '' ;
binmode STDIN;
binmode STDOUT;

# Read in the whole file, 1 Mb should cover it. It'll end w/o error
# when it hits the end of stdin

read (STDIN, $input, 1000000);

# strip off the uncompressed header. Don't need the swap  in variable
# but with my limited perl knowledge I didn't want to mess with the
# original prog much

$tempin = substr($input,41);
$input=$tempin;

# While the input string still has stuff in it - this works
# because the inflate function modifies input to have whatever
# is left in it after the first complete chunk is uncompressed
# (the 4000 byte section Dan mentioned)

while (length($input) > 0 )
{
# Initialize a new structure - this needs to be done for each call
# to inflate. die gracefully if it fails from lack of memory, whatever

$x = inflateInit()
  or die "Cannot create a inflation stream\n" ;

# call inflate. It returns
#  $output - the decompressed stream, which will be max 4000 bytes
#  $status - a return code. 0 is success,
#                           1 is end of data (i.e. you didn't send
#                             a complete stream, and it expected
#                             more data
#  $input  - contains the remainder of $input after the stream that
#            got decoded into $output
#

($output, $status) = $x->inflate(\$input) ;

# write $output to STDOUT. each write gets appended to the previous, so
# you eventually build up the whole file. The first write still needs
# to be modified to strip off the header that is in the file after
# decompression

print $output;

# Gracefully exit if the decode failed. This means something bad
# happened. This is how I discovered that I had both the compressed
# and encrypted data in one file. The compressed stuff worked but the
# encrypted didn't.

if ($status < 0 or $status > 2) {
die "bogus second header";
}
}
#------- Cut Here --------------
In a previous message to me, you wrote: >I have also been able to uncompress the ARX NIDS data but I needed to do some >tricks to get it to work. This is what I did... its pretty ugly!! 
>
>It turns out that NWS has split the NIDS file into 4000 byte blocks and then >compressed the data using the zlib compress utility. Then the compressed >blocks are sent out sequentially as part of the whole product. This means >you have separate the blocks to decode. > >The problem is that the uncompress utility doesn't work that effectively. >The problem is you don't know the size of the compressed blocks. The block >sizes are buried in the SBN header which is masked off by the LDM. So you >have to guess what the block sizes are. The uncompress utility will return >the size of the uncompressed data but won't reveal the size of the compressed >data. > >What I've done is to forward through the data 1000 bytes at a time and then >try to uncompress the data. If the function returns an error, I increment >through the data a byte at a time until I get a valid return on the >uncompression. This is just plain ugly.... 
>
>What I'm looking for is a simpler solution!! If there is another utility >that gives me the compressed block sizes, this would make my job much easier! 
>
>BTW, I read where the uncompress utility will uncompress all the data if the >file is memory mapped??? Otherwise, it only works a block at a time. 
>
>So Pete, can you forward the Perl script so I can check what functions are >used?? 
>
>Dan.... from cloudy Chicago!!


--
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
^ Systems Programmer               V Madison,         WI     53706    ^
^                                  V      poker@xxxxxxxxxxxxxxx       ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
^ University of Wisconsin-Madison  V       262-0166 (Fax)             ^
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+
  • 2000 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the decoders archives: