[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
20021114: AWIPS Full Court Press
- Subject: 20021114: AWIPS Full Court Press
- Date: Thu, 14 Nov 2002 09:37:24 -0700
>From: Tom Whittaker <address@hidden>
>Organization: SSEC
>Keywords: 200211141624.gAEGO8L23840 NOAAPORT GOES compress PNG wavelet
Tom, et. al.,
>At the bottom of this posting (echoed in the AWIPSinfo list this morning):
>
>http://isl715.nws.noaa.gov/addl/foa/telecon/Awp_1112.htm
>
>is a commitment ("pending community approval" -- whatever that means) to
>compress the GOES data into one channel, but not on any specified date.
> Thought you'd be interested.
I have been in contact with Tom Renkevens, Barb Hickman, and others
about compression techniques off and on since the October MUG meeting.
I provided statistics on the efficiency of our lossless PNG compression
technique that they found interesting. Since you may be interested in
these, I include them below. If not, stop reading now :-)
From: Tom Yoksas <address@hidden>
Date: Sat, 26 Oct 2002 19:08:58 -0600
To: "Thomas Renkevens" <address@hidden>
cc: Frederick R Mosher <address@hidden>,
Kevin Schrab <address@hidden>, address@hidden,
address@hidden
Subject: 20021026: Unidata lossless PNG compression of NOAAPORT image
products (correction)
Tom, Fred, Kevin,
...
I finally was able to free up enough time to compile some statistics on
the lossless PNG technique we use to compress NOAAPORT GINI images we
send in our IDD.
The following numbers represent NOAAPORT GINI imagery received between
23 Z Friday, October 25 to 23 Z Saturday, October 26. We compress the
images received in NOAAPORT before injecting them into our IDD. The
compressed images are uncompressed upon receipt at downstream hosts.
PNG Compression
GINI Product Mean Std Product Description
---------------------+------+-------+------------------------
AK-NATIONAL_VIS 2.285 0.352 Alaska Nat 8 km 0.65 um
AK-NATIONAL_WV 6.911 0.795 Alaska Nat 8 km 6.8 um
AK-NATIONAL_IR 3.001 0.409 Alaska Nat 8 km 10.7 um
AK-REGIONAL_VIS 1.718 0.162 Alaska Reg 2 km 0.65 um
AK-REGIONAL_3.9 1.962 0.040 Alaska Reg 8 km 3.9 um
AK-REGIONAL_WV 3.576 0.057 Alaska Reg 16km 6.8 um
AK-REGIONAL_IR 2.077 0.024 Alaska Reg 8 km 10.7 um
AK-REGIONAL_12.0 2.085 0.025 Alaska Reg 8 km 12.0 um
HI-NATIONAL_VIS 3.599 0.865 Hawaii Nat 14km 0.65 um
HI-NATIONAL_WV 8.813 2.105 Hawaii Nat 14km 6.8 um
HI-NATIONAL_IR 4.377 1.006 Hawaii Nat 14km 10.7 um
HI-REGIONAL_VIS 1.878 0.257 Hawaii Reg 1 km 0.65 um
HI-REGIONAL_3.9 2.616 0.375 Hawaii Reg 4 km 3.9 um
HI-REGIONAL_WV 3.947 0.511 Hawaii Reg 8 km 6.8 um
HI-REGIONAL_IR 2.688 0.342 Hawaii Reg 4 km 10.7 um
HI-REGIONAL_12.0 2.656 0.333 Hawaii Reg 4 km 12.0 um
PR-NATIONAL_VIS 2.341 0.922 P Rico Nat 8 km 0.65 um
PR-NATIONAL_WV 5.401 1.921 P Rico Nat 8 km 6.8 um
PR-NATIONAL_IR 2.920 1.172 P Rico Nat 8 km 10.7 um
PR-REGIONAL_VIS 2.159 0.533 P Rico Reg 1 km 0.65 um
PR-REGIONAL_3.9 2.858 0.756 P Rico Reg 4 km 3.9 um
PR-REGIONAL_WV 4.049 1.070 P Rico Reg 8 km 6.8 um
PR-REGIONAL_IR 2.918 0.781 P Rico Reg 4 km 10.7 um
PR-REGIONAL_12.0 2.896 0.763 P Rico Reg 4 km 12.0 um
PNG Compression
GINI Product Mean Std Product Description
---------------------+------+-------+------------------------
EAST-CONUS_VIS 1.947 0.173 East Conus 1 km 0.65 um
EAST-CONUS_3.9 2.203 0.069 East Conus 4 km 3.9 um
EAST-CONUS_WV 4.274 0.083 East Conus 8 km 6.8 um
EAST-CONUS_IR 2.544 0.974 East Conus 4 km 10.7 um
EAST-CONUS_12.0 2.515 0.054 East Conus 4 km 12.0 um
WEST-CONUS_VIS 1.857 0.113 West Conus 1 km 0.65 um
WEST-CONUS_3.9 2.295 0.052 West Conus 4 km 3.9 um
WEST-CONUS_WV 4.391 0.109 West Conus 8 km 6.8 um
WEST-CONUS_IR 2.518 0.082 West Conus 4 km 10.7 um
WEST-CONUS_12.0 2.487 0.069 West Conus 4 km 12.0 um
PNG Compression
GINI Product Mean Std Product Description
---------------------+------+-------+------------------------
NHEM-COMP_VIS 1.760 0.161 Nth Hemi 24km 0.65 um
NHEM-COMP_3.9 2.011 0.133 Nth Hemi 24km 3.9 um
NHEM-COMP_WV 3.429 0.245 Nth Hemi 24km 6.8 um
NHEM-COMP_IR 1.983 0.136 Nth Hemi 24km 10.7 um
NHEM-COMP_12.0 2.013 0.138 Nth Hemi 24km 12.0 um
NHEM-MULTICOMP_VIS 2.050 0.139 MultiSat 24km 0.65 um
NHEM-MULTICOMP_WV 2.809 0.111 MultiSat 24km 6.8 um
NHEM-MULTICOMP_IR 1.815 0.019 MultiSat 24km 10.7 um
SUPER-NATIONAL_VIS 1.583 0.018 SuprNatnl 8 km 0.65 um
SUPER-NATIONAL_3.9 1.959 0.040 SuprNatnl 8 km 3.9 um
SUPER-NATIONAL_WV 4.286 0.044 SuprNatnl 8 km 6.8 um
SUPER-NATIONAL_IR 2.051 0.019 SuprNatnl 8 km 10.7 um
SUPER-NATIONAL_12.0 2.073 0.018 SuprNatnl 8 km 12.0 um
SUPER-NATIONAL_CTP 6.148 0.703 SuprNatnl 8 km Cld Top Pres
SUPER-NATIONAL_LI 6.853 0.823 SuprNatnl 8 km Lifted Index
SUPER-NATIONAL_PW 6.789 0.818 SuprNatnl 8 km Precip Water
SUPER-NATIONAL_SFC-T 6.463 0.761 SuprNatnl 8 km Sfc Temp
As a comparison, the same numbers for various composites we create from
the NOAAPORT NEXRAD level III products is:
PNG Compression
GINI Product Mean Std Product Description
---------------------+------+-------+------------------------
RADAR_n0r 9.248 1.611 1 km N0R composite
RADAR_n1p 63.142 5.198 2 km N1P composite
RADAR_ntp 13.549 1.075 4 km NTP composite
Values in the 'Mean' column represent the average compression ratio
over the 24 hour ingest period, and values in the 'Std' column
represent the standard deviation of those compression ratios.
As you would expect for satellite imagery, VIS images compress the
least and water vapor compress the most. Compression ratios for
satellite products (e.g., CTP, LI, PW, and SFC-T) are all good.
I hope these numbers are useful.
Tom
From: Tom Yoksas <address@hidden>
Date: Mon, 28 Oct 2002 12:21:38 -0700
To: "Barbara Hickman" <address@hidden>
cc: Thomas Renkevens <address@hidden>,
Howard Carney <address@hidden>,
Stephen Gilbert <address@hidden>,
Dorothy Brown <address@hidden>,
Patrick Otero <address@hidden>,
Brian Callicott <address@hidden>,
Patti Skews <address@hidden>, address@hidden.
edu
Subject: 20021028: [Fwd: 20021026: Unidata lossless PNG compression of
NOAAPO
RT image products (correction)]
Hi Barb,
re: PNG compression techniques used at the UPC
>Do you have any similar information on the timing delays per product
>introduced with PNG?
No, but my feeling is that they can't be very long. The system we use
for NOAAPORT ingest (one we built ourselves) has very little buffering
(4 MB), and that buffering is used by all 4 channels. This means that
-- at a maximum --, 3.5 T1s of data is pouring into that 4 MB buffer,
and no products are being lost during ingest. To me, this says that
that the on-the-fly PNG compression that we are running (we are reading
image products out of the ingest card buffer and compressing the image
bytes as they become available) does not add any significant time delay
to the injection of any NOAAPORT product into our IDD.\
I don't think that you can use our time delay experience to judge what
the impact would be on your end, however. I can put together some
numbers of how long it takes to PNG compress full NOAAPORT images
stored in files if you like, but this will take a little time to pull
together. I can more quickly put together some numbers on the time it
takes to PNG compress images stored in McIDAS AREA files.
My opinion is that our PNG compression will be significantly faster
than wavelet compression techniques since it is a lot less CPU
intensive.
>I'm also curious as to where/how you obtained the
>compression utility package (i.e. cost & source)
The PNG stuff is like Zlib (and uses Zlib): it is available
free-of-charge. The PNG Home Page (which is linked off of the Zlib
Home Page) is:
http://www.libpng.org/pub/png/
The following entry in the PNG FAQ page sheds light on PNGs availability:
Q: I thought you said PNG was patent-free--what about Stac, PKWARE, Apple,
etc.?
A: The PNG image format was designed in 1995 specifically in response to
the patent problems with the LZW algorithm used in GIF. To the best of
the PNG group's knowledge, PNG was then--and still
is--completely patent-free.
However, the fact that is possible to implement PNG without infringing
any known patents certainly does not imply that it is impossible to
implement it without infringing any patents. Patents are a minefield,
and several are known to be closely related to PNG technologies:
o Stac, LZ77 sorted hash, deflate algorithm
o PKWARE, whatever, deflate algorithm
o Apple, alpha mask images
>and how it is tuned/configured for your usage,
We use a standard build of PNG in our compression routines. We have
not done any additional tuning (yet) even though this would likely
improve the compression ratios that we can get for the NOAAPORT
products. Customization is an important point. The PNG Home Page
gives several examples of how people have created custom filters that
greatly improve compression of specific types of data. Some of the
results listed are spectacular.
Our compression application simply reads the input, writes out the
unaltered header and then compresses the image data and appends it to
the output. I include a schematic of the procedure at the end of this
note.
>type of platform on which it is run,
Our PNG compression/uncompression routines run on all of the OSes we
support for our applications:
Compaq Tru64 5.1
FreeBSD 4.[57]
HP-UX B 11.00
IBM AIX 4.3, 5.1
RedHat Linux 6.x, 7.x (we will look at 8.x soon)
SGI IRIX, IRIX64 6.5.x
Sun Solaris 5.[6789] (SPARC and x86)
I have not tried this on Windows, but I can't say for sure that it
would work there, but given that PNG compressed images are supported in
Microsoft IE, I have to imagine that it would.
>how the platform is equipped, etc.
The code runs on the wide variety of platforms in use at our sites.
This ranges from 200 Mhz Linux PCs with 64 MB of RAM to Sun Enterprise
E450s and Sun Blade 1000, etc.
>Your overall numbers look
>consistent with our generalized product findings (Vis - least, WV -
>most, product subsat, resolution, and coverages by channel also play
>into the rates achieved, etc.) although the mean ratios do look
>significantly better.
I am glad that our results are consistent.
>Right now, though, NWS is specifically working with zlib since they
>believe that it is more readily avaialable and will have less impact to
>both government AWIPS users and commercial NOAAport users.
PNG uses Zlib compression techniques. Both packages are freely
available and run on a very wide range of OSes (lots more than we
support our applications on). Before dismissing PNG in favor of a
Zlib-only approach, I would hope that you read through the PNG Home
Page information and consider that display of PNG compressed imagery is
supported in virtually all current web browsers. As far as the last
comment goes, our sites can strip off the uncompressed header of the
PNG compressed files we send in our IDD and look at the images in their
web broswer. This makes it extremely easy for them to take quick looks
at imagery without even uncompressing it.
Our PNG compression applications effectively do the following:
NOAAPORT GINI:
+-----------------------+ +-------------------------+
| Header | | Header |
+-----------------------+ +-------------------------+
| | | |
| | PNG | |
| Uncompressed | =====> | Compressed |
| Data | | Data |
| | | |
| | | |
| | +-------------------------+
| |
| |
| |
| |
| |
+-----------------------+
McIDAS AREA:
+-----------------------+ +-------------------------+
| Header | | Header |
+-----------------------+ +-------------------------+
| Nav | | Nav |
+-----------------------+ PNG +-------------------------+
| Cal | =====> | Cal |
+-----------------------+ +-------------------------+
| AUX | | AUX |
+-----------------------+ +-------------------------+
| | | Comment Cards |
| | +-------------------------+
| | | |
| | | |
| Uncompressed | | Compressed |
| Data | | Data |
| | | |
| | | |
| | +-------------------------+
| |
| |
| |
| |
| |
+-----------------------+
| Comment Cards |
+-----------------------+
We keep both the GINI and AREA header information uncompressed so that
the header information can be used without uncompressing it first. For
instance, a McIDAS ADDE server for either only needs the header
information to do things like IMGLISTs. The data only has to be
uncompressed when it is needed by things like IMGDISP, IMGCOPY,
IMGREMAP, etc.
Tom
From address@hidden Mon Oct 28 18:37:29 2002
To: "Barbara Hickman" <address@hidden>
cc: Thomas Renkevens <address@hidden>,
Howard Carney <address@hidden>,
Stephen Gilbert <address@hidden>,
Dorothy Brown <address@hidden>,
Patrick Otero <address@hidden>,
Brian Callicott <address@hidden>,
Patti Skews <address@hidden>, address@hidden
Subject: 20021028: [Fwd: 20021026: Unidata lossless PNG compression of
NOAAPORT image products (correction)]
Hi Barb,
>Do you have any similar information on the timing delays per product
>introduced with PNG?
Here are some numbers on how long it took to PNG compress a series of
GOES-East VIS images stored in McIDAS AREA files. The files all contain
1-byte images created from a single scan from 18:15Z on 28 Oct 02301 as
follows:
The largest image, stored in AREA3000, was created from the GER/GENHEM01V
dataset on the SATEPS machine with IP address 140.90.195.64:
IMGCOPY GER/GENHEM01V MYDATA/IMAGES.3000 TIME=18:15 SIZE=SAME STYPE=VISR
MAG=1 -2
IMGLIST MYDATA/IMAGES.3000 FORM=EXP
Image file directory listing for:MYDATA/IMAGES
Pos Satellite/ Date Time Center Res (km) Image_Size
sensor Lat Lon Lat Lon
--- ------------- ------------ -------- ---- ---- ----- -----
------------
3000 G-8 IMG 28 OCT 02301 18:15:00 14 72
Band: 1 0.65 um Visible - Cloud Cover 1.06 1.15 7308 x
6928
proj: 0 created: 2002302 5752 memo: RT GVAR
type:VISR cal type:BRIT
offsets: data= 2816 navigation= 256 calibration= 0 auxillary= 0
doc length: 0 cal length: 0 lev length: 0 PREFIX= 0
valcod: 0 zcor: 0 avg-smp: N
start yyddd: 2002301 start time:181515 start scan: 351
lcor: 2805 ecor: 9049 bytes per pixel: 1 ss: 70
Resolution Factors (base=1): Line= 1.0 Element= 2.0
IMGLIST: done
Each smaller image was created from the local VISR copy of GER/GENHEM01V
by blowing down by a factor of 2**n:
AREA IMGCOPY command that created AREA file
-----+---------------------------------------------------------------------
3001 IMGCOPY MYDATA/IMAGES.3000 MYDATA/IMAGES.3001 MAG=-2 -2 SIZE=SAME
3002 IMGCOPY MYDATA/IMAGES.3000 MYDATA/IMAGES.3002 MAG=-4 -4 SIZE=SAME
3003 IMGCOPY MYDATA/IMAGES.3000 MYDATA/IMAGES.3003 MAG=-8 -8 SIZE=SAME
3004 IMGCOPY MYDATA/IMAGES.3000 MYDATA/IMAGES.3004 MAG=-16 -16 SIZE=SAME
The following is a list of the times it took to compress the images
using the AREA PNG compression routine I wrote:
AREA AREA size PNG Size Time to compress (mm:ss)
-----+----------+---------+------------------------
3000 50632960 28858185 1:32
3001 12660672 7905475 0:15
3002 3167580 2124720 0:03
3003 795700 567738 0.00 (faster than 1 second)
3004 202112 151957 0.00 (faster than 1 second)
File sizes are in bytes.
The compression routine was run on a Sun E450 here in the UPC offices.
The E450 has 4 450 Mhz SPARC Ultra II processors. The compression
routine does not take advantage of multiple processors, so the
numbers should be the same for a single processor 450 Mhz Ultra
II SPARC that is not running other jobs.
You can see that the time it takes to compress images does not scale
linearly in input AREA file size.
I hope that these numbers are useful. The are, at least, interesting.
Tom
Tom
