[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: question about "new" EASE-Grids



Hi Mary Jo:

Sorry for the long delay, your email came while I was on vacation, and Im still catching up.

On 10/28/2011 10:50 AM, Mary Jo Brodzik wrote:

John, You may remember me, I think we met at a netCDF meeting a year or two ago.

yes, of course. Also you live on Twin Sisters Road.


I work at NSIDC. We've been producing data in the so-called "EASE-Grid" for a number of data sets over the years, mostly it's stored as flat, binary arrays, we have one pilot data set in netCDF-CF, but there's motivation to use netCDF-CF for new data, and to deliver old data using on-the-fly capabilities to format as netCDF-CF.

The reason I'm writing you is the the mention in the cf-satellite posting below of the page at

http://www.unidata.ucar.edu/software/netcdf-java/reference/StandardCoordinateTransforms.html

that you maintain.

The original EASE-Grid has accumulated the inevitable warts over the years that I'm currently trying to address with a revised, "EASE-Grid-2.0", definition. The biggest problem with EASE-Grid was that it was used to store mainly satellite data what was of course referenced to the WGS84 datum, but the EASE-Grid projection was spherical (based on 1924 International Authalic Sphere). This has caused loads of trouble for software handling it, and so the most significant change I'm proposing for EASE-Grid-2.0 is to use the WGS84 ellipsoid for the projections. EASE-Grid is actually a family of projections: Northern and Southern Lambert Azimuthal Equal-Area for polar regions and cylindrical equal-area for mid- and low-latitude regions.

As I promote the use of EASE-Grid-2.0, I'm trying to keep track of which software packages may or may not have trouble with it, and I notice that on your Standard Coordinate Transformation pages, you list lambert_azimuthal_equal_area but have a note that it's a spherical earth model only. I also can't find any cylindrical projections on that page.

you are right that the code we have for lambert_azimuthal_equal_area is only for spherical earth, and we dont have cylindrical equal-area. We do have ellipsoidal code for both from proj4 that we can convert, but i dont do it until i have some test files. If you want to send me some netcdf files with these projections, i will add when i get the chance, under the condition that you will double check that i did it correctly.


I'm wondering what are the implications of these omissions. Would it be correct to say that the netCDF java implementation of projection transforms does not support any of the EASE-Grid-2.0 definitions, and possibly not even the original (spherical) cylindirical equal-area definition?

yes, thats how it stand now.


Thanks for any insight/comments,
Mary Jo

Regards,
John


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mary Jo Brodzik, Special Projects Lead, 303-492-8263
NSIDC/CIRES, Univ. of Colo. at Boulder, 449 UCB, Boulder, CO 80309-0449
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

---------- Forwarded message ----------
Date: Tue, 25 Oct 2011 10:14:55 +0200
From: Peter Miu <address@hidden>
To: 'Martin Raspaud' <address@hidden>
Cc: "address@hidden" <address@hidden>
Subject: Re: [cf-satellite] cf-satellite Digest, Vol 14, Issue 1

Morning Martin,

Sorry to hear that you were at the user conference but were not able to attend the presentation I gave. I've attached the presentation for information, and I welcome feedback on the subject matter
from subscribers of this mailing list.

The southMostLine, eastMostPixel, northMostLine, westMostPixel global attributes are defined to inform users who are used to using these parameters of the subarea ordered. This is particularly useful when users order a lot of data as it reminds them of the contents of the data. Yes I agree that this is not essential but adding meta-data like this even though it may only be useful to some users adds value to the data. Knowing the needs of your core users will help determine what type of meta-data to specify
and your input is very useful.

I agree that implementing the HRV in 2 arrays is a good way to organise the data. This avoids a lot of cells being 'empty' which would be the case if the data was stored in one array.
We will be revising the MSG15 NetCDF data set to include more data
stored in the native format so this will give us an opportunity to implement the HRV channel
as well.

Regarding your lat lon comment, Unidata provides Standard Coordinate Transforms to
Geo-locating data onto a earth projection, see below:

http://www.unidata.ucar.edu/software/netcdf-java/reference/StandardCoordinateTransforms.html

This removes the need to store the lat and lon arrays in your data if you are using Unidata tools to visualise the data. For users who don't use these tools, it would require them to map these transforms to lat and lon i.e. they need to do extra programming to visualise the data. As I've mentioned before, usability of the data for all users is very important in our development. So even though the lat and lon uses space in the NetCDF file, it is
provided for usability reasons.

If you take a look at the presentation, we proposed the concept of tailoring NetCDF data sets to the user's needs. Whether the lat and lon arrays, coordination transform or any other method of geo-locating the data is included in the NetCDF file or not can be addressed here. We just need to get enough users requesting the implementation of tailored NetCDF files (hint :o).

Regarding the band discussion and taking Tom's reinforcing Email into account (Thanks Tom ... might see you next year in Boulder if you are there to discuss more CF conventions with you) I think I understand what bands are now. Creating a 3D array indexed by a band or channel with lat and lon is not a problem. For me, I think the channel/band should be an coordinate variable so that it can be treated like 'level', "altitude" or "pressure level" by the Unidata tools for visualisation. I'm also in favour of the standard_name="channel" rather than "band" as this is clearer to me, of course this is only my input to this discussion.

I'm don't know who you have been talking to at EUMETSAT regarding bands but now I know this is a user need, I can address this in our Format Advisory Group. Thanks for drawing my attention to this.

In case anyone reading this thread have problems in understanding some of the term we are using. I've found the following tutorial to help understanding the basic NetCDF concepts and I hope this is useful.

http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-tutorial.pdf

Regards,

Pete.

-----Original Message-----
From: Martin Raspaud [mailto:address@hidden]
Sent: Monday, October 24, 2011 5:08 PM
To: Peter Miu
Cc: address@hidden
Subject: Re: [cf-satellite] cf-satellite Digest, Vol 14, Issue 1

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 24/10/11 15:35, Peter Miu wrote:
Hi Martin,

Hi Pete,

Thanks for the fast feedback.  Before I answer your questions, I would
just like to let you know that the EUMETSAT Data Centre is currently
planning the implementation NetCDF formats as the common delivery format for all orderable data sets from the Archive.

A pilot has already been performed for the level 1B and 2 products
from the Metop-A ASCAT instrument and these are available for user feedback under the following URL:

http://gsics.eumetsat.int/thredds/ascat.html

The ASCAT NetCDF products were also presented to users in this year's EUMETSAT user conference held in Oslo.

The development of usable NetCDF formats is our top priority as we
want to ensure that all users (novice and expert) can immediately use
our data so it is very important that guidelines regarding best
practises, filename conventions, CF-Conventions and standards are
followed.  A Format Advisory Group has been set up at EUMETSAT to
discuss these guidelines with a view to publishing them so that users
are aware how our data is organised and also to propose updates to the
CF-conventions for satellite data as work is needed here (hence the
existence of this mailing list! :o)

I was at the EUMETSAT conference this year :)

Here are the answers to your questions:

- Why are southMostLine, eastMostPixel, northMostLine, westMostPixel, and numberOfChannels dimensions and not attributes ?

Apart from the HRV channel, these values remain constant for all
channels in the data sets so we can be specify them as global
attributes.  If the channel arrays store different 'sub-area' then we
would need these value to be stored as variable attributes for each channel array. This is not the case for our NetCDF data set i.e. all channel arrays hold the same area for the same time.

Note, the NetCDF file follows the Unidata recommendation for
identifying the coordinates of the data in the channel arrays
(Coordinate System Object Model - Current Encodings). Longitude and latitude arrays exist in the data set and their contents are used to geo-locating the pixel counts for each channel onto a projection.

See: http://www.unidata.ucar.edu/software/netcdf-java/CDM/

Ok. I don't really understand the interest of having these dimensions/attributes though, since I guess you will have to define lon/lats anyway for each different resolution...

- - How would you deal with the inclusion of the HRV channel ?

We have not implemented the HRV channel in our NetCDF data sets as GSICS does not use this channel for calibration.

To implement this channel in our NetCDF data sets, we would just need
to add another longitude and latitude array specifying the location of each HRV pixel counts and a 2D array holding the HRV pixel counts.

Sounds good, but the HRV has 2 sub-areas: would you have a different variable for each or would you group the in one big channel ? (We implemented the latter, since the netcdf compression works well in this
case)

- - Are the longitudes and latitudes values different from what a "vertical perspective" with right parameters would provide ? If they do, why ? If they don't, why not include a grid-mapping definition ?

Not sure what you mean here, perhaps the following explain it:

For the pixel count in ch1( y, x ), it is located at latitude lat( y, x ) and longitude lon( y, x ). By assigning the lat and lon as the coordinates system for any channel array, applications know how to geo-locate its contents.

  float lat(y,x);
  float lon(y,x);
  int ch1(y,x);
  ch1:coordinates="lat lon";

  int ch2(y,x);
  ch2:coordinates="lat lon";

My question was if the lon/lats differ from what one would get from running proj.4 with parameters like
+proj=geos +lon0=0 +a=... etc.
If one would get the same lon/lat data, then I would recommend to record also these parameters in a grid mapping variable. In this case, the lon/lats arrays would even become optional.

- - On the cf-satellite list, we have long been discussing "band"s. Did you consider using those ?

Sorry, I don't know what bands are.

Well, this is a bit sad :(. We have been working quite a lot on this list toward having band variables that would correctly describe satellite channels (or bands). My colleagues and I have tried to drag Eumetsat's attention to this list and the work done here on several occasions. Tom Whittaker was also at the Eumetsat conference last year in Cordoba to present CF conventions and the possibility to apply these to satellite. But apparently that was not enough...

You can read more on the band variable and related questions here:
http://www.unidata.ucar.edu/mailing_lists/archives/cf-satellite/2011/thread.html

- - Did you consider grouping channels in 3d arrays instead of having them separated ?

I'm not sure what value a 3D arrays would give to the data.
Applications like the Unidata can plot the arrays on top of each other if needed.

As mentioned above, we want our data to be immediately usable by all
users and by following guidelines, we add values to the data as the
contents can be examined and plotted by existing free NetCDF applications without any further processing by the users.

- - Could you provide in the file calibration coefficients needed to convert radiances to reflectances and brightness temperatures ?

Yes we can add any variable attribute to the channel arrays to support the needs of our users.
The power of NetCDF is as long as we don't remove or reorganise any
attributes and/or arrays, existing users and applications will not
'break' by the change.  The only concern here is we need to ensure
that a standard convention is followed for representing measurements
that can be stored in different units (e.g. temperature can be stored as Celsius, Fahrenheit, Kelvin, ... ). What unit is used should be based on the needs of the 'core' users and this unit with a standard name should be included in the CF-conventions.


That would be nice :) Thanks a lot.

Best regards,
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Red Hat - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOpX9pAAoJEBdvyODiyJI4mC0IAK/6m0+TWMtaVcRChVGLJ9VX
OY4OsSZQ+dceW4+YWGymBRUInUWChURdvy4eLCX9lPiz2TgOXbJkyvVcuK2iULtp
6oYTX+Fn3g3LhDM118rlYe5CApgFcg0bA62hqwXoM0ksrdQ6cbt1tz6aSV1kpkbL
4tCJrmLLen8mcYF3fSsH1xpKli5NUXEOwcf4jDWg9ieQAwic4RuD8bky1rshK3NS
DDJeVjkKETE9VBMsseupWmptuWWG7zzFH9qsMatHjn8tYixTfezwaURu3C8n00Fm
ugwHLO1uUkCb+jK7Ga1Kikftg7hFlATMTgLd1WRpZHyO0LqbbrgiLmvRHbsVYV0=
=waU8
-----END PGP SIGNATURE-----