[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20000414: Access to SSEC realtime imagery database Unidata sites (cont.)

>From: UW SSEC Data Center <address@hidden>
>Organization: Space Science Data Center
>Keywords: 200003232140.OAA18829 McIDAS-X Unidata-Wisconsin ADDE FTP SSEC 
>unirec imagery

Dee,

>Sorry I did not get back to you today, yesterday was a real zoo and I am just 
>reading that mail today.  Jerry and I were talking about how we needed to
>get back to you earlier in the week.  I will respond to your questions below.

OK, no problem.  I have been pretty busy myself.

re: summary of .EDU requests from SSEC Data Center
>February 2000 - 4 requests, 2 processed, 1 rejected (data not in agreed
>time frame), 1 sent to CIMSS as he wanted products not raw data and
>CIMSS produce them.  2 requests processed = 222 scenes for a total of
>6.4GB of data
>
>March 2000 - 6 requests, 1 rejected
>             5 proceeded = 139 scenes for 2.4 GB of data
>April 2000 - 2 requests so far this month
>             1,004 scenes and 10.4 GB of data
>
>*note: a scene is a time period it could be 1-5 bands (areas) of
>imagery The numbers above are only for satellite data, there were 10
>days of conventional data processed.

OK, thanks.

re: increasing data sent in Unidata-Wisconsin datastream
>Increasing the data on the LDM should not be a problem as defined as
>long as a new machine is in place.  You should contact your program
>manager because this may affect funding.

I guess that I don't understand your last comment.  My intention is
that we would purchase the machine and do the initial setup here at the
UPC.  No offense, but it seems like SSEC always buys underpowered,
overpriced IBM machines.  Much more of a machine can be purchased with
the same dollars from independent vendors.  How would _our_ purchase of
a machine affect funding?

re:
>> In order to effectively survey the Unidata community about what they
>> would like to see in the Unidata-Wisconsin broadcast (i.e. data sent
>> through the IDD on a regular basis), I need to know if they will be
>> allowed to access higher resolution (temporal and spatial) imager and
>> sounder data directly from the realtime SSEC database?  The reason for
>> this is my perception that users will ask for a lot more data on a
>> regular basis than they may, in fact, need IF there is no possibility
>> of getting additional data in another manner.  I believe that the
>> imagery appetite of the great majority of Unidata sites is quite
>> modest.  There will be a few sites, however, that will want higher
>> resolution VIS imagery for their areas of the country and/or other
>> imagery/sounder bands for research projects.

>You are right that users will ask for the "whole world" at full
>resolution. Some of the .EDU requests asked for full res for several
>days and even though they were questions about the amount they were
>asking for they thought that is what they wanted.

I think that the majority of users really don't have a good handle on
how large satellite images can be.  This has been demonstrated to me by
conversations with a user about possible modifications to the contents
of the UW stream.  It is only after users with "eyes too big for their
stomachs" try to grab what they request that they realize that they
probably have data storage issues and/or internet bandwidth issues.  I
am hoping to educate my users on this subject by using the statistics
that I have gathered on how well (or poorly as the case may be) AREA
files in the UW stream are compressed.  As a quick, non-final heads-up
on this, I offer the following:

  Current         AREA         Delta encoded  PNG compressed    PNG
Image Sector    file size      product size   product size    Savings
---------------+--------------+--------------+---------------+--------
GOES-West VIS    2439056        2145447        1525458         619989
GOES-West IR     2439056        1309143         990694         318449
GOES-West H2O     613426         225347         156641          68706
GOES-East VIS    2422256        2142377        1545774         596603
GOES-East IR     2422256        1388884        1068211         320673
GOES-East H2O     608371         231733         167498          64235
Floater I         607776         397478         307993          89485
Floater II        607776         399856         311895          91863
---------------+--------------+--------------+---------------+--------
Sub Total       12159973        8240265        6074164        2179003

Mollweide IR      225088         163798         104013          59785
Mollweide H2O     225088         138423          70699          67724
Antarctic IR      308509         187221         135232          51989
MDR               225088          47667           4655          43012
---------------+--------------+--------------+---------------+--------
Sub Total         983773         537109         314599         222510

---------------+--------------+--------------+---------------+--------
Totals          13143746        8777374        6388763        2401513

You can see from this listing that the delta encoding compression
scheme currently used for the UW datastream is very poor for VIS
images.  PNG compression techniques (PNG is based on zlib) do a lot
better, but are not as good as I had hoped they would be.

By the way, I compared delta encoding, PNG, compress, gzip, and bzip2
compression techniques on images currently conveyed in the UW stream
for a one week period.  PNG was the overall best at compressing the
imagery, but bzip2 typically did better for images that contained a lot
of "black" (e.g., VIS images very early in the morning or late in the
evening).  I will be contacting Chad about setting up some test
transmissions (IDD EXP stream transfers) of CIMSS products from SSEC to
the UPC sometime next week.  I realize that Chad is very busy with the
May upgrade, so I am not anticipating rapid turn around...

>After the data was
>processed they found out it was too much to handle.  DC-ops wised up
>quickly so now when they get orders like, every conus at full res for a
>week, they process one day and wait and see if the user can deal with
>it before continuing.

Sounds like a wise move!

>Allowing Unidata to get and serve its user higher res data is certainly
>possible.  Previously you mentioned a purchasing a machine,  "a dual
>600 (or faster) Mhz Pentium III with a couple GB of RAM and
>significant(like up to 100 GB of Ultra SCSI-2) disk." and have your
>users go to this machine to et the higher res data.

Right.  Now the machine would probably be a dual 800 Mhz machine.  Time
marches on; machines get faster for the same price.

>Like Tom mentioned
>we bounced around whether it is what should be done or if your users
>should have direct access to our servers.  Initially I was for giving
>direct access, actually I was the one to bring it up, but I know think
>that we should start with building the high res data on the new machine
>and have your users come to that. HOw we get the data to that machine
>needs to be discussed,.but not this week.

OK.  I originally proposed the idea of staging the data to a single
machine to try and isolate the traffic and offload the burden from the
Data Center staff.

>SO....Jerry and I would like to talk to you next week to brainstorm on
>this. Pick a day and time that is good for you and we will plan a phone
>call.

I will be in my office from 1:30 until about 5:30 this afternoon (our
time), so if you and Jerry have some time, I would love to chat.  I
will not be in tomorrow (last skiing week of the season here in
Colorado), but I will be in all day Friday.  Again, my number is
303.497.8642.

Tom