[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20021114: AWIPS Full Court Press



>From: Tom Whittaker <address@hidden>
>Organization: SSEC
>Keywords: 200211141624.gAEGO8L23840 NOAAPORT GOES compress PNG wavelet

Tom, et. al.,

>At the bottom of this posting (echoed in the AWIPSinfo list this morning):
>
>http://isl715.nws.noaa.gov/addl/foa/telecon/Awp_1112.htm
>
>is a commitment ("pending community approval" -- whatever that means) to 
>compress the GOES data into one channel, but not on any specified date. 
>  Thought you'd be interested.

I have been in contact with Tom Renkevens, Barb Hickman, and others
about compression techniques off and on since the October MUG meeting.
I provided statistics on the efficiency of our lossless PNG compression
technique that they found interesting.  Since you may be interested in
these, I include them below.  If not, stop reading now :-)


  From: Tom Yoksas <address@hidden>
  Date: Sat, 26 Oct 2002 19:08:58 -0600
  To: "Thomas Renkevens" <address@hidden>
  cc: Frederick R Mosher <address@hidden>,
     Kevin Schrab <address@hidden>, address@hidden,
     address@hidden
  Subject: 20021026: Unidata lossless PNG compression of NOAAPORT image 
products (correction)
  
  Tom, Fred, Kevin,
  
  ...

  I finally was able to free up enough time to compile some statistics on
  the lossless PNG technique we use to compress NOAAPORT GINI images we
  send in our IDD.
  
  The following numbers represent NOAAPORT GINI imagery received between
  23 Z Friday, October 25 to 23 Z Saturday, October 26.  We compress the
  images received in NOAAPORT before injecting them into our IDD.  The
  compressed images are uncompressed upon receipt at downstream hosts.
  
                        PNG Compression
  GINI Product          Mean   Std     Product Description
  ---------------------+------+-------+------------------------
  AK-NATIONAL_VIS       2.285  0.352   Alaska Nat 8 km  0.65 um
  AK-NATIONAL_WV        6.911  0.795   Alaska Nat 8 km  6.8  um
  AK-NATIONAL_IR        3.001  0.409   Alaska Nat 8 km 10.7  um
  AK-REGIONAL_VIS       1.718  0.162   Alaska Reg 2 km  0.65 um
  AK-REGIONAL_3.9       1.962  0.040   Alaska Reg 8 km  3.9  um
  AK-REGIONAL_WV        3.576  0.057   Alaska Reg 16km  6.8  um
  AK-REGIONAL_IR        2.077  0.024   Alaska Reg 8 km 10.7  um
  AK-REGIONAL_12.0      2.085  0.025   Alaska Reg 8 km 12.0  um
  HI-NATIONAL_VIS       3.599  0.865   Hawaii Nat 14km  0.65 um
  HI-NATIONAL_WV        8.813  2.105   Hawaii Nat 14km  6.8  um
  HI-NATIONAL_IR        4.377  1.006   Hawaii Nat 14km 10.7  um
  HI-REGIONAL_VIS       1.878  0.257   Hawaii Reg 1 km  0.65 um
  HI-REGIONAL_3.9       2.616  0.375   Hawaii Reg 4 km  3.9  um
  HI-REGIONAL_WV        3.947  0.511   Hawaii Reg 8 km  6.8  um
  HI-REGIONAL_IR        2.688  0.342   Hawaii Reg 4 km 10.7  um
  HI-REGIONAL_12.0      2.656  0.333   Hawaii Reg 4 km 12.0  um
  PR-NATIONAL_VIS       2.341  0.922   P Rico Nat 8 km  0.65 um
  PR-NATIONAL_WV        5.401  1.921   P Rico Nat 8 km  6.8  um
  PR-NATIONAL_IR        2.920  1.172   P Rico Nat 8 km 10.7  um
  PR-REGIONAL_VIS       2.159  0.533   P Rico Reg 1 km  0.65 um
  PR-REGIONAL_3.9       2.858  0.756   P Rico Reg 4 km  3.9  um
  PR-REGIONAL_WV        4.049  1.070   P Rico Reg 8 km  6.8  um
  PR-REGIONAL_IR        2.918  0.781   P Rico Reg 4 km 10.7  um
  PR-REGIONAL_12.0      2.896  0.763   P Rico Reg 4 km 12.0  um
  
                        PNG Compression
  GINI Product          Mean   Std     Product Description
  ---------------------+------+-------+------------------------
  EAST-CONUS_VIS        1.947  0.173   East Conus 1 km  0.65 um
  EAST-CONUS_3.9        2.203  0.069   East Conus 4 km  3.9  um
  EAST-CONUS_WV         4.274  0.083   East Conus 8 km  6.8  um
  EAST-CONUS_IR         2.544  0.974   East Conus 4 km 10.7  um
  EAST-CONUS_12.0       2.515  0.054   East Conus 4 km 12.0  um
  WEST-CONUS_VIS        1.857  0.113   West Conus 1 km  0.65 um
  WEST-CONUS_3.9        2.295  0.052   West Conus 4 km  3.9  um
  WEST-CONUS_WV         4.391  0.109   West Conus 8 km  6.8  um
  WEST-CONUS_IR         2.518  0.082   West Conus 4 km 10.7  um
  WEST-CONUS_12.0       2.487  0.069   West Conus 4 km 12.0  um
  
                        PNG Compression
  GINI Product          Mean   Std     Product Description
  ---------------------+------+-------+------------------------
  NHEM-COMP_VIS         1.760  0.161   Nth Hemi   24km  0.65 um
  NHEM-COMP_3.9         2.011  0.133   Nth Hemi   24km  3.9  um
  NHEM-COMP_WV          3.429  0.245   Nth Hemi   24km  6.8  um
  NHEM-COMP_IR          1.983  0.136   Nth Hemi   24km 10.7  um
  NHEM-COMP_12.0        2.013  0.138   Nth Hemi   24km 12.0  um
  NHEM-MULTICOMP_VIS    2.050  0.139   MultiSat   24km  0.65 um
  NHEM-MULTICOMP_WV     2.809  0.111   MultiSat   24km  6.8  um
  NHEM-MULTICOMP_IR     1.815  0.019   MultiSat   24km 10.7  um
  SUPER-NATIONAL_VIS    1.583  0.018   SuprNatnl  8 km  0.65 um
  SUPER-NATIONAL_3.9    1.959  0.040   SuprNatnl  8 km  3.9  um
  SUPER-NATIONAL_WV     4.286  0.044   SuprNatnl  8 km  6.8  um
  SUPER-NATIONAL_IR     2.051  0.019   SuprNatnl  8 km 10.7  um
  SUPER-NATIONAL_12.0   2.073  0.018   SuprNatnl  8 km 12.0  um
  SUPER-NATIONAL_CTP    6.148  0.703   SuprNatnl  8 km Cld Top Pres
  SUPER-NATIONAL_LI     6.853  0.823   SuprNatnl  8 km Lifted Index
  SUPER-NATIONAL_PW     6.789  0.818   SuprNatnl  8 km Precip Water
  SUPER-NATIONAL_SFC-T  6.463  0.761   SuprNatnl  8 km Sfc Temp
  
  As a comparison, the same numbers for various composites we create from
  the NOAAPORT NEXRAD level III products is:
  
                        PNG Compression
  GINI Product          Mean   Std     Product Description
  ---------------------+------+-------+------------------------
  RADAR_n0r             9.248   1.611  1 km N0R composite
  RADAR_n1p            63.142   5.198  2 km N1P composite
  RADAR_ntp            13.549   1.075  4 km NTP composite
  
  Values in the 'Mean' column represent the average compression ratio
  over the 24 hour ingest period, and values in the 'Std' column
  represent the standard deviation of those compression ratios.
  
  As you would expect for satellite imagery, VIS images compress the
  least and water vapor compress the most.  Compression ratios for
  satellite products (e.g., CTP, LI, PW, and SFC-T) are all good.
  
  I hope these numbers are useful.
  
  Tom

  From:      Tom Yoksas <address@hidden>
  Date:      Mon, 28 Oct 2002 12:21:38 -0700
  To:        "Barbara Hickman" <address@hidden>
  cc:        Thomas Renkevens <address@hidden>,
             Howard Carney <address@hidden>,
             Stephen Gilbert <address@hidden>,
             Dorothy Brown <address@hidden>,
             Patrick Otero <address@hidden>,
             Brian Callicott <address@hidden>,
             Patti Skews <address@hidden>, address@hidden.
            edu
  Subject:   20021028: [Fwd: 20021026: Unidata lossless PNG compression of 
NOAAPO
            RT image products (correction)] 
  
  Hi Barb,
  
  re: PNG compression techniques used at the UPC
  
  >Do you have any similar information on the timing delays per product
  >introduced with PNG?
  
  No, but my feeling is that they can't be very long.  The system we use
  for NOAAPORT ingest (one we built ourselves) has very little buffering
  (4 MB), and that buffering is used by all 4 channels.  This means that
  -- at a maximum --, 3.5 T1s of data is pouring into that 4 MB buffer,
  and no products are being lost during ingest.  To me, this says that
  that the on-the-fly PNG compression that we are running (we are reading
  image products out of the ingest card buffer and compressing the image
  bytes as they become available) does not add any significant time delay
  to the injection of any NOAAPORT product into our IDD.\
  
  I don't think that you can use our time delay experience to judge what
  the impact would be on your end, however.  I can put together some
  numbers of how long it takes to PNG compress full NOAAPORT images
  stored in files if you like, but this will take a little time to pull
  together.  I can more quickly put together some numbers on the time it
  takes to PNG compress images stored in McIDAS AREA files.
  
  My opinion is that our PNG compression will be significantly faster
  than wavelet compression techniques since it is a lot less CPU
  intensive.
  
  >I'm also curious as to where/how you obtained the
  >compression utility package (i.e. cost & source)
  
  The PNG stuff is like Zlib (and uses Zlib): it is available
  free-of-charge.  The PNG Home Page (which is linked off of the Zlib
  Home Page) is:
  
  http://www.libpng.org/pub/png/
  
  The following entry in the PNG FAQ page sheds light on PNGs availability:
  
  Q: I thought you said PNG was patent-free--what about Stac, PKWARE, Apple, 
etc.?
  
    A: The PNG image format was designed in 1995 specifically in response to
       the patent problems with the LZW algorithm used in GIF. To the best of
       the PNG group's knowledge, PNG was then--and still
       is--completely patent-free. 
  
       However, the fact that is possible to implement PNG without infringing
       any known patents certainly does not imply that it is impossible to
       implement it without infringing any patents. Patents are a minefield,
       and several are known to be closely related to PNG technologies: 
  
       o Stac, LZ77 sorted hash, deflate algorithm 
       o PKWARE, whatever, deflate algorithm 
       o Apple, alpha mask images 
  
  >and how it is tuned/configured for your usage,
  
  We use a standard build of PNG in our compression routines.  We have
  not done any additional tuning (yet) even though this would likely
  improve the compression ratios that we can get for the NOAAPORT
  products.  Customization is an important point.  The PNG Home Page
  gives several examples of how people have created custom filters that
  greatly improve compression of specific types of data.  Some of the
  results listed are spectacular.
  
  Our compression application simply reads the input, writes out the
  unaltered header and then compresses the image data and appends it to
  the output.  I include a schematic of the procedure at the end of this
  note.
  
  >type of platform on which it is run,
  
  Our PNG compression/uncompression routines run on all of the OSes we
  support for our applications:
  
    Compaq Tru64 5.1
    FreeBSD 4.[57]
    HP-UX B 11.00
    IBM AIX 4.3, 5.1
    RedHat Linux 6.x, 7.x (we will look at 8.x soon)
    SGI IRIX, IRIX64 6.5.x
    Sun Solaris 5.[6789] (SPARC and x86)
    
  I have not tried this on Windows, but I can't say for sure that it
  would work there, but given that PNG compressed images are supported in
  Microsoft IE, I have to imagine that it would.
  
  >how the platform is equipped, etc.
  
  The code runs on the wide variety of platforms in use at our sites.
  This ranges from 200 Mhz Linux PCs with 64 MB of RAM to Sun Enterprise
  E450s and Sun Blade 1000, etc.
  
  >Your overall numbers look
  >consistent with our generalized product findings (Vis - least, WV -
  >most, product subsat, resolution, and coverages by channel also play
  >into the rates achieved, etc.) although the mean ratios do look
  >significantly better.
  
  I am glad that our results are consistent.
  
  >Right now, though, NWS is specifically working with zlib since they
  >believe that it is more readily avaialable and will have less impact to
  >both government AWIPS users and commercial NOAAport users.
  
  PNG uses Zlib compression techniques.  Both packages are freely
  available and run on a very wide range of OSes (lots more than we
  support our applications on).  Before dismissing PNG in favor of a
  Zlib-only approach, I would hope that you read through the PNG Home
  Page information and consider that display of PNG compressed imagery is
  supported in virtually all current web browsers.  As far as the last
  comment goes, our sites can strip off the uncompressed header of the
  PNG compressed files we send in our IDD and look at the images in their
  web broswer.  This makes it extremely easy for them to take quick looks
  at imagery without even uncompressing it.
  
  Our PNG compression applications effectively do the following:
  
  NOAAPORT GINI:
  
  
         +-----------------------+                +-------------------------+
         |        Header         |                |          Header         |
         +-----------------------+                +-------------------------+
         |                       |                |                         |
         |                       |      PNG       |                         |
         |     Uncompressed      |     =====>     |        Compressed       |
         |         Data          |                |           Data          |
         |                       |                |                         |
         |                       |                |                         |
         |                       |                +-------------------------+
         |                       |
         |                       |
         |                       |
         |                       |
         |                       |
         +-----------------------+
  
  
  McIDAS AREA:
  
         +-----------------------+                +-------------------------+
         |        Header         |                |          Header         |
         +-----------------------+                +-------------------------+
         |         Nav           |                |            Nav          |
         +-----------------------+     PNG        +-------------------------+
         |         Cal           |    =====>      |            Cal          |
         +-----------------------+                +-------------------------+
         |         AUX           |                |            AUX          |
         +-----------------------+                +-------------------------+
         |                       |                |      Comment Cards      |
         |                       |                +-------------------------+
         |                       |                |                         |
         |                       |                |                         |
         |     Uncompressed      |                |        Compressed       |
         |         Data          |                |           Data          |
         |                       |                |                         |
         |                       |                |                         |
         |                       |                +-------------------------+
         |                       |
         |                       |
         |                       |
         |                       |
         |                       |
         +-----------------------+
         |    Comment Cards      |
         +-----------------------+
  
  
  We keep both the GINI and AREA header information uncompressed so that
  the header information can be used without uncompressing it first.  For
  instance, a McIDAS ADDE server for either only needs the header
  information to do things like IMGLISTs.  The data only has to be
  uncompressed when it is needed by things like IMGDISP, IMGCOPY,
  IMGREMAP, etc.
  
  Tom
  
  From address@hidden Mon Oct 28 18:37:29 2002
  To: "Barbara Hickman" <address@hidden>
  cc: Thomas Renkevens <address@hidden>,
     Howard Carney <address@hidden>,
     Stephen Gilbert <address@hidden>,
     Dorothy Brown <address@hidden>,
     Patrick Otero <address@hidden>,
     Brian Callicott <address@hidden>,
     Patti Skews <address@hidden>, address@hidden
  Subject: 20021028: [Fwd: 20021026: Unidata lossless PNG compression of 
NOAAPORT image products (correction)] 
  
  Hi Barb,
  
  >Do you have any similar information on the timing delays per product
  >introduced with PNG?
  
  Here are some numbers on how long it took to PNG compress a series of
  GOES-East VIS images stored in McIDAS AREA files.  The files all contain
  1-byte images created from a single scan from 18:15Z on 28 Oct 02301 as
  follows:
  
  The largest image, stored in AREA3000, was created from the GER/GENHEM01V
  dataset on the SATEPS machine with IP address 140.90.195.64:
  
  IMGCOPY GER/GENHEM01V MYDATA/IMAGES.3000 TIME=18:15 SIZE=SAME STYPE=VISR 
MAG=1 -2
  IMGLIST MYDATA/IMAGES.3000 FORM=EXP
  Image file directory listing for:MYDATA/IMAGES
   Pos Satellite/         Date       Time      Center      Res (km)   Image_Size
       sensor                                 Lat  Lon    Lat   Lon
   --- -------------  ------------  --------  ---- ----  ----- ----- 
------------
  3000  G-8 IMG       28 OCT 02301  18:15:00    14   72
      Band: 1   0.65 um Visible - Cloud Cover             1.06  1.15  7308 x 
6928
       proj:    0 created: 2002302   5752  memo: RT GVAR
       type:VISR     cal type:BRIT
       offsets:  data=    2816 navigation=  256 calibration=    0 auxillary=    0
       doc length:   0   cal length:   0   lev length:   0 PREFIX=   0
       valcod:          0 zcor:  0 avg-smp: N
       start yyddd: 2002301  start time:181515  start scan:  351
       lcor: 2805  ecor:  9049  bytes per pixel: 1  ss: 70
       Resolution Factors (base=1):   Line=    1.0   Element=    2.0
  IMGLIST: done
   
  Each smaller image was created from the local VISR copy of GER/GENHEM01V
  by blowing down by a factor of 2**n:
  
  AREA  IMGCOPY command that created AREA file
  -----+---------------------------------------------------------------------
  3001  IMGCOPY MYDATA/IMAGES.3000 MYDATA/IMAGES.3001 MAG=-2 -2 SIZE=SAME
  3002  IMGCOPY MYDATA/IMAGES.3000 MYDATA/IMAGES.3002 MAG=-4 -4 SIZE=SAME
  3003  IMGCOPY MYDATA/IMAGES.3000 MYDATA/IMAGES.3003 MAG=-8 -8 SIZE=SAME
  3004  IMGCOPY MYDATA/IMAGES.3000 MYDATA/IMAGES.3004 MAG=-16 -16 SIZE=SAME
  
  
  The following is a list of the times it took to compress the images
  using the AREA PNG compression routine I wrote:
  
  AREA  AREA size  PNG Size  Time to compress (mm:ss)
  -----+----------+---------+------------------------
  3000  50632960   28858185  1:32
  3001  12660672    7905475  0:15
  3002   3167580    2124720  0:03
  3003    795700     567738  0.00  (faster than 1 second)
  3004    202112     151957  0.00  (faster than 1 second)
  
  File sizes are in bytes.
  
  The compression routine was run on a Sun E450 here in the UPC offices.
  The E450 has 4 450 Mhz SPARC Ultra II processors.  The compression
  routine does not take advantage of multiple processors, so the
  numbers should be the same for a single processor 450 Mhz Ultra
  II SPARC that is not running other jobs.
  
  You can see that the time it takes to compress images does not scale
  linearly in input AREA file size.
  
  I hope that these numbers are useful.  The are, at least, interesting.
  
  Tom

Tom