[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #ZRG-608224]: SSEC neartime data archive

Hi Dave,

re: FNEXRAD/UNIWISC announcement sent to multiple email lists

> Several years ago the University firewalled mail servers other than their 
> own, including
> one that I maintained for myself and our students on campus. My Unidata 
> registration at
> the time used that since-firewalled email address, and I stopped getting 
> email notices
> from the idvusers and ldm-users email lists (which would be the two that I 
> had most
> likely joined). I changed my Unidata registration last year, but it’s 
> possible that the
> new email address didn’t migrate to those mailing lists, because I still 
> haven’t
> received anything via those lists in a very long time.

OK.

re:
> My functioning email address is address@hidden. The
> non-functional address, which is probably what those email lists are still 
> trying to
> use, is address@hidden.

I just subscribed you (Dave Dempsey <address@hidden> to the idvusers and
ldm-users email lists.  Interestingly, I did not see your old email address
in the list of subscribed users to either of these lists.


re:
> Some of the satellite imagery in “decoded/gempak/areas/…” goes back 30 days, 
> but not
> all.

Yes, on motherlode this is true.  atm.ucar.edu, on the other hand, has been
receiving the FNEXRAD and UNIWISC datastreams from our Amazon EC2 instance
for some time now, so you can find longer nearterm archives on it.

re:
> For example:
> 
> GOES-13/4km imagery:
> 
> Does go back 30 days: 0.65, 10.7, 13.3, 3.9, and 6.5 um. (Doesn’t 0.65 um 
> imagery date
> back to GOES-12 rather than GOES-13? See 
> http://cimss.ssec.wisc.edu/goes/calibration
> /bandwidths.html<http://cimss.ssec.wisc.edu/goes/calibration/bandwidths.html>)
> Does not go back 30 days (only 5 days): 0.63 um.

Yup.  The file used in the PNG-compression process was updated to a much more
current version.  One effect of this is to correct the wavelength channel for
both GOES-East and West imagery.

By the way, the five days marks the switchover to the new FNEXRAD/UNIWISC
datastreams last Friday evening.  This record will grow day by day.

re:
> GOES-13/1km imagery:
> Does go back 30 days: nothing
> Does not go back 30 days (only 5 days): 0.63 um.

Yup.

re:
> GOES-15/4km imagery:
> 
> Does go back 30 days: 10.7, 13.3, 3.9, and 6.5 um.
> Normally goes back 30 days but is missing everything after 0200Z on March 22 
> through
> today: 0.65 um. (Again, doesn’t 0.65 um imagery date back to GOES-12 rather 
> than
> GOES-15?

There are current GOES-West (GOES-15) images for all imager channels. The
problem is that the wavelength changed for the VIS channel, so there are
two directory hierarchies that comprise the full set of images.  You can
verify that there are current images in the IDV by get a listing of
the absolute times for GOES-West or GOES-East images in the RTIMAGES
dataset from either/both adde.ucar.edu (which is motherlode) and
unidata2-new.ssec.wisc.edu.  Again, atm.ucar.edu has been receiving the
new FNEXRAD and UNIWISC products for some time now, so its set of
"current" images is much longer than on motherlode or unidata2-new.

re:
> Note also that it was the fact these these same images are missing from SSEC’s
> archive that prompted this exchange in the first place. Looks like Unidata 
> didn’t get
> them, either!)

I don't think that this is the case.  Please see my explanation above.

re:
> Doesn’t go back 30 days (only 5 days): 0.62 um. (This is listed as 0.625 um, 
> which would
> round to 0.63 um, in 
> http://cimss.ssec.wisc.edu/goes/calibration/bandwidths.html.)

Yup.

re:
> MOLLWEIDE IR imagery:
> Goes back 5 days, plus two days in February (25 and 26).
> 
> MOLLWEIDE WV imagery:
> GOES back 5 days, plus Feb. 25 through March 3 (but not March 4 through March 
> 21).

These image types are not new, but the are significantly different from
what they were previously (higher resolution and larger size).  The are much
more useful now than before.  Try loading one or more of these in the globe
view in the IDV from atm.ucar.edu.

re: saving data in AREA format from the IDV
> Saving in McIDAS AREA format is _not_ supported in the IDV ... I wish that
> it was!

Ditto!

re:
> I had in mind simply saving the file that the IDV accesses from a remote 
> server, in its
> original format. That would use the IDV as a glorified file transfer program.

Ditto, ditto!!

re: converting the images using 'pnga2area'

> Yep, that does the trick all right! I think that occurred to me at some point 
> but it
> must have been late and I didn’t follow through to check it.

Very good.

re:
> This morning I am able to use the IDV to display both a compressed and an 
> uncompressed
> MCIDAS Area file downloaded from motherlode > 
> decoded/gempak/areas/GOES-15/4km/…. Not
> sure what the problem was yesterday.

I'm glad to hear that things are working (and that I wasn't imagining things :-)

re:
> Note that I can display the downloaded files only if they are local files, 
> not (the same
> files) accessed via our RAMADDA server (http://virga.sfsu.edu:8080). (This is 
> an issue
> that’s been in the Unidata “attend  to" queue for several years now.)

Yes, RAMADDA/THREDDS does not server satellite imagery nearly as well as
McIDAS ADDE.  When it/they will is a good question, but I wouldn't count on
it being soon.

re: cost of operating in the cloud
> Hmmm. I wonder what the implications are for the vision articulated in 
> Unidata’s latest
> 5-year NSF proposal? Guess the cloud won’t be including Amazon (or other 
> commercial
> clouds?) any time soon?

Here are a couple of comments I sent to another user:

  Since you mentioned it, operating a motherlode-clone in the
  Amazon cloud would be very expensive -- an estimate using volumes
  being served off of motherlode about a year ago was in the
  neighborhood of $250K (!) per year.  The instance we are currently
  working with should end up costing us between $3-3.6K per year.
  The problem is you pretty much never know what something costs
  until you get the bill.

  We are investigating other clouds (MS Azure and Google) to see
  if/where we can save money AND increase services.

  One other thing: it appears that intra-cloud transfers of data
  is very cheap.  This should mean that we could operate server
  instances like motherlode in the cloud and provide data to other
  machines running in the same cloud for little to no money (at
  least for the time being).  This setup might work well if Unidata
  community members moved their "stuff" to the cloud.

re: moving the Unidata-Wisconsin archive to a new machine

> OK, good to know. In the meantime, I’ve revised my auto-download script for 
> satellite
> images to access motherlode (via wget) as an alternative to 
> unidata2.ssec.wisc.edu.

Very good.

re:
> By the way, how come motherlode provides access to 1km GOES-13 (GOES-East) 
> visible
> images but not 1km GOES-15 (GOES-West) visible images? (And the 1km images 
> that it does
> have only go back 5 days—presumably for ADDE service, which only goes back 50 
> hours as
> viewed by the IDV? Or maybe for disk storage space reasons?)

The reason is historical: I started creating GOES-East/South composites for
the Unidata-Wisconsin datastream a number of years ago.  One step in the 
North-South
compositing process was the generation of a 1km VIS sector that was never sent
in the IDD (but it was served via ADDE from the machine on which it was
created).  At the time I did this work, I asked some users if they were
interested in North-South composites from GOES-West, and they said that
they weren't that interested.  After sending out the message about the new 
products
that would be included in UNIWISC, I got feedback that indicates that the
interest in North-South and high resolution VIS composites from GOES-West
would be welcome.  Creation of these composites products is on my list of 
things to
do.  Stay tuned...

re:
> As usual, thanks for the thorough and accessible responses. As usual, for 
> customer
> service you set the platinum standard. :-)

Thanks, that is a very nice comment!  I do try...

To catch you up, here is the announcement that I sent out to the 4 email lists
last Thursday morning.  Please note the inclusion of a 3-hourly global WV
and thermal IR composite!


Users of Unidata IDD FNEXRAD and UNIWISC datastream products:

At or shortly after 0 UTC on Saturday, March 21 (18:00 MDT Friday),
the contents-of the Internet Data Distribution FNEXRAD (NEXRAD Level III
national composites) and UNIWISC (aka McIDAS, satellite imagery sectors)
datastreams will be updated with new products.

The previous source (creation point) for the FNEXRAD products was
motherlode.ucar.edu, and the source (creation point) for the UNIWISC
products was unidata2.ssec.wisc.edu.  Both datastreams are now being
created in a 64-bit CentOS 6.5 VM in the Amazon EC2 cloud.

To: address@hidden, address@hidden,
        address@hidden, address@hidden
Date: Thu, 20 Mar 2014 10:03:46 -0600
From: Unidata User Support <address@hidden>
Subject: [ldm-users] 20140320: Impending changes to IDD FNEXRAD and UNIWISC 
datastreams

The following is a brief overview of the contents of each datastream
with special emphesis on new or altered products:

FNEXRAD - NEXRAD Level III composites:

Product   Res   Cod  Type                Form   Status
--------+------+----+-------------------+------+----------------
National  1 km  N0R  base reflectivity   GINI   existing product
National  2 km  N1P  1-hour precip.      GINI   existing product
National  4 km  NTP  storm total precip. GINI   existing product
National  6 km  N0R  base reflectivity   GINI   existing product
National 10 km  RCM  base reflectivity   GINI   existing product
National  1 km  N0R  base reflectivity   GRIB2  existing product

National  1 km  DHR  base reflectivity   GINI   *new* product
National  1 km  DVL  dig. vert. liq H2O  GINI   *new* product
National  1 km  EET  enhanced echo tops  GINI   *new* product
National  1 km  HHC  hybrid hydro. class GINI   *new* product

Notes:

- all products are updated every 5 minutes
- all GINI images are PNG compressed
- further information including sample displays of these Level III
  composites can be found in:
  http://www.unidata.ucar.edu/data/radar.html#fnexrad


UNIWISC (aka MCIDAS) - Satellite Image Sectors:

Product                                         Res   Form   Status          
----------------------------------------------+------+-----+-----------------
GOES-East Visible                               4 km  PNGA   existing product
GOES-East Shortwave Infrared                    4 km  PNGA   existing product
GOES-East Water Vapor                           4 km  PNGA   existing product
GOES-East Thermal Infrared                      4 km  PNGA   existing product
GOES-East Longwave Infrared (CO2)               4 km  PNGA   existing product 
GOES-West Visible                               4 km  PNGA   existing product
GOES-West Shortware Infrared                    4 km  PNGA   existing product
GOES-West Water Vapor                           4 km  PNGA   existing product
GOES-West Thermal Infrared                      4 km  PNGA   existing product
GOES-West Longwave Infrared (CO2)               4 km  PNGA   existing product
Antarctic Thermal Infrared composite            5 km  PNGA   existing product
Manually digitized radar                       10 km  PNGA   existing product

Global Mollweide Water Vapor composite         30 km  PNGA   higher resolution
Global Mollweide Thermal Infrared composite    30 km  PNGA   higher resolution
GOES-East HiRes Visible                         1 km  PNGA   *new* product
GOES-East/West NH Visible composite            10 km  PNGA   *new* product
GOES-East/West NH Shortwave Infrared composite 10 km  PNGA   *new* product
GOES-East/West NH Water Vapor composite        10 km  PNGA   *new* product
GOES-East/West NH Thermal Infrared composite   10 km  PNGA   *new* product
GOES-East/West NH Longwave Infrared composite  10 km  PNGA   *new* product
Arctic Visible composite                        4 km  PNGA   *new* product
Arctic Shortwave Infrared composite             4 km  PNGA   *new* product
Arctic Water Vapor composite                    4 km  PNGA   *new* product
Arctic Thermal Infrared composite               4 km  PNGA   *new* product
Arctic Longwave Infrared composite              4 km  PNGA   *new* product
Antarctic Visible composite                     5 km  PNGA   *new* product
Antarctic Water Vapor composite                 5 km  PNGA   *new* product
Global Rectilinear Water Vapor composite       20 km  PNGA   *new* product
Global Rectilinear Thermal Infrared composite  20 km  PNGA   *new* product

Notes:

- update cycles vary by product
- all images are in a PNG-compressed McIDAS AREA format
- Global Mollweide WV and IR composites are now higher resolution and
  larger in size
- further information including sample displays of these satellite image
  sectors can be found in:
  http://www.unidata.ucar.edu/data/datastream.html


Potential impacts to end-users:

- the Global Mollweide composite products will be 4x the size/double
  the resolution of the ones that have previously been included

  This will likely require updates to users' automated processing
  routines (e.g., IDV bundles, McIDAS BATCH/MCB files, shell scripts, etc.).

- McIDAS ADDE dataset names on Unidata-maintained servers (e.g.,
  ADDE.UCAR.EDU, ATM.UCAR.EDU, ADDE.SSEC.WISC.EDU, LEAD4.UNIDATA.UCAR.EDU)
  will be updated to the following:

Satellite Image Sectors:

RTIMAGES/ANTARCTICVIS  Antarctic Visible composite
RTIMAGES/ANTARCTICWV   Antarctic Water Vapor composite
RTIMAGES/ANTARCTIC     Antarctic Thermal Infrared composite
RTIMAGES/ANTARCTICIR   Antarctic Thermal Infrared composite
RTIMAGES/ARCTIC-VIS    Arctic Visible composite
RTIMAGES/ARCTIC-IRS    Arctic Shortwave Infrared composite
RTIMAGES/ARCTIC-WV     Arctic Water Vapor composite
RTIMAGES/ARCTIC-IR     Arctic Thermal Infrared composite
RTIMAGES/ARCTIC-IRL    Arctic Longwave Infrared composite
RTIMAGES/GE-1KVIS      GOES-East HiRes Visible
RTIMAGES/GE-VIS        GOES-East Visible
RTIMAGES/GE-39         GOES-East Shortwave Infrared
RTIMAGES/GE-WV         GOES-East Water Vapor
RTIMAGES/GE-IR         GOES-East Thermal Infrared
RTIMAGES/GE-CO2        GOES-East Longwave Infrared (CO2)
RTIMAGES/GW-VIS        GOES-West Visible
RTIMAGES/GW-39         GOES-West Shortware Infrared
RTIMAGES/GW-WV         GOES-West Water Vapor
RTIMAGES/GW-IR         GOES-West Thermal Infrared
RTIMAGES/GW-CO2        GOES-West Longwave Infrared (CO2)
RTIMAGES/GEW-VIS       GOES-East/West NH Visible composite
RTIMAGES/GEW-39        GOES-East/West NH Shortwave Infrared composite
RTIMAGES/GEW-WV        GOES-East/West NH Water Vapor composite
RTIMAGES/GEW-IR        GOES-East/West NH Thermal Infrared composite
RTIMAGES/GEW-CO2       GOES-East/West NH Longwave Infrared composite
RTIMAGES/GLOBAL-WV     Global Rectilinear Water Vapor composite
RTIMAGES/GLOBAL-IR     Global Rectilinear Thermal Infrared composite
RTIMAGES/MOLL-WV       Global Mollweide Water Vapor composite
RTIMAGES/MOLL-IR       Global Mollweide Thermal Infrared composite
RTIMAGES/MDR           Manually digitized radar

GLOBAL/WVCOMP          Global Rectilinear Water Vapor composite
GLOBAL/IRCOMP          Global Rectilinear Thermal Infrared composite
GLOBAL/MOLL-WV         Global Mollweide Water Vapor composite
GLOBAL/MOLL-IR         Global Mollweide Thermal Infrared composite

NEXRAD Level III National Composites:

NEXRCOMP/1KN0R-NAT     1 km N0R US Base Reflectivity Composite
NEXRCOMP/2KN1P-NAT     2 km N1P US 1-hr Precip. Composite
NEXRCOMP/4KNTP-NAT     4 km NTP US Storm Total Precip. Composite
NEXRCOMP/6KN0R-NAT     6 km N0R National Base Reflectivity Composite
NEXRCOMP/10KRCM-NAT    10 km RCM National Composite Radar
NEXRCOMP/1KDHR-NAT     1 km DHR US Digital Hybrid Reflectivity Composite
NEXRCOMP/1KDVL-NAT     1 km DVL US Digital Vert. Int. Liquid H2O Composite
NEXRCOMP/1KEET-NAT     1 km EET US Enhanced Echo Tops Composite
NEXRCOMP/1KHHC-NAT     1 km HHC US Hybrid Hydrometeor Class. Composite

Notes:

- RTIMAGES/ANTARCTICIR serves the same data as RTIMAGES/ANTARCTIC;
  RTIMAGES/ANTARCTICIR is the new, preferred name

- RTIMAGES/GLOBAL-WV serves the same data as GLOBAL/WVCOMP;
  RTIMAGES/GLOBAL-WV is the new, preferred name

- RTIMAGES/GLOBAL-IR serves the same data as GLOBAL/IRCOMP;
  RTIMAGES/GLOBAL-IR is the new, preferred name

- GLOBAL/MOLL-WV serves the same data as RTIMAGES/MOLL-WV;
  RTIMAGES/MOLL-WV is still the preferred name

- GLOBAL/MOLL-IR serves the same data as RTIMAGES/MOLL-IR;
  RTIMAGES/MOLL-IR is still the preferred name

- the GLOBAL/WVCOMP and GLOBAL/IRCOMP images that have been served
  by the Unidata-operated ADDE server adde.ucar.edu will be reduced
  from previous instances by a factor of about 5 (20 km versus 4-5 km
  previously)

Please send any questions or comments to Unidata Datastream Support
<address@hidden>

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ZRG-608224
Department: Support Datastream
Priority: Normal
Status: Closed