Unidata Users Committee
2-3 April 2012

Committee Members
Acronym List

Members Attending:
Tom Whittaker, Univ of Wisc-Madison-Space Science & Engineering Center, Chair
Martin Baxter, Central Michigan Univ
Anne T. Case-Hanks, Univ of Louisiana, Monroe
Jennifer Collins, Univ of South Florida
Gerry Creager, Texas A&M
Steven Lazarus, Florida Institute of Technology
Kevin Tyle, Univ of Albany
Bart Geerts, Univ of Wyoming

Student Rep:
Stefan Cecelski, Univ of Maryland

NCEP Reps:
Becky Cosgrove (CONDUIT) - virtual participation
Michelle Mainelli (GEMPAK-NAWIPS/AWIPS II)

DeSouza Award Recipient:
David Knight, Univ of Albany

Richard Signell

Staff Attending:

Tina Campbell
John Caron
Julien Chastang
Ethan Davis
Doug Dirks
Ben Domenico
Steve Emmerson
Ginger Emery
Ward Fisher
Dennis Heimbigner

Marcos Hermida
Yuan Ho
Michael James
Robb Kambic
Linda Miller
Terry Mitchell
Jennifer Oxelson
Mohan Ramamurthy
Mike Schmidt
Jeff Weber
Tom Yoksas

Monday, 2 April 2012

Next Meeting

The next Users Committee meeting will be 4-5 October 2012. It was suggested that it be a joint meeting with the Policy Committee. Date Amendment: The Users Committee will meet in Boulder, Colorado on September 17-18, with the 18th being the joint meeting day with the Policy Committee)


There were two actions from the previous meeting:

ACTION: TomWhittaker will send an invitation to the Unidata community to submit DeSouza nominations in the December timeframe - Done

ACTION: The UserComm will talk with community members about their data needs and the impending CONDUIT survey when they are contacting them for their site surveys - Done

Policy Committee Report

Set up a WRF sub committee-Gary Lackmann, Brian Colle et al. Unidata will have a two-hour session at the next WRF meeting. The UPC also did a talk at the VAPOR session during AMS
Improved communications and interactions with MMM are still needed.

Director's Report – Mohan Ramamurthy

Mohan's Slides

Highlights Include:

Two new software engineers, Marcos Hermida, who is working on THREDDS and Ward Fisher, who is working on netCDF support for Windows platforms. UPC continues to search for a Systems Administrator and a NOAA-funded software engineer to work on the OPeNDAP-Unidata collaborative project (OPULS). A summer Code intern is also being sought.

The Russell L. DeSouza Award is being presented at this meeting to David Knight, who has been active in the Unidata community for many years via governing committees, workshops, and helping universities gain access to real-time NLDN lightning data.

Dr. Thomas Bogdan took over as President of UCAR in January 2012.

Work is progressing to create an action/implementation plan associated with the Strategic Plan. The plan was formally adopted by the Policy Committee in early January. The plans will become the basis of the next proposal to be submitted in early 2013.

Staff allocation by project is nearing desired levels.

Real-time data flows are feeding about 500 machines at 230 sites running LDM-6. UPC's IDD cluster relays about 650 downstream connections with the average output of 9.3 TB/day.

There is 55.4 TBytes of data per year brought in via ADDE/OPeNDAP. Motherlode handles this data and maintains about a two week rollover, but UPC is not an archive site.

Software development infrastructure is being upgraded. The source code for various projects (CDM/TDS/IDV/LDM, with VISAD planned) is moving to Github. Development teams are moving towards using more community and collaboration features of Jira, Redmine. Development teams are starting to implement Agile development processes ("sprints").

During the WRF workshop session, IDV played a prominent role. Closer ties to WRF are warranted and from a poll of WRF users: IDV, VAPOR, NCL are the top 3 visualization tools.

IDV 3.0u2 was released in February. The IDV developers are working on time syncing, axis scale/labeling interface, enhanced support for ensemble output. The IDV has been moved to Github as fully open source. IDV and McIDAS-V teams are working collaboratively between the two projects.

No major changes to NAWIPS/GEMPAK or McIDAS-X

AWIPS II (D2D, NSHARP, GFE) A 64 bit EDEX server will be available winter 2012-13. Several UPC staff had a field trip to the local WFO at the Skagg's Bldg. They are conducting side-by-side demonstrations between AWIPS I and AWIPS II to measure performance and functionality.

There are now over 40,000 users in 200 countries in 2400 academic institutions in the Unidata community.

EarthCube: 111 white papers were submitted and 10 awards were granted. See: http://earthcube.ning.com/ Unidata is involved in some of the projects, but not the lead on any of them. (see presentation for additional details)

Yuan Ho and Tom Yoksas conducted a two-day workshop at California University of Pennsylvania on 16-17 February. The workshop funding was provided by CUP.

The Africa Meningitis Project is now a 3-year Google.org funded project under a UCAR Africa Initiative. The IDV is being used to generate displays using the TIGGE ensemble data from ECMWF. (Check out the slide 24 for more information)

Unidata had many visitors at both the AMS Student Conference and the AMS Annual Meeting in New Orleans. A great deal of interest was demonstrated in AWIPS II and IDV. UPC staff were joined by NCEP folks to demonstrate the capabilities of AWIPS II.

Community Equipment Awards: Unidata has made 61 awards in past 9 years, totaling approximately $1million. A small number of proposals were received this year to compete for the community award funding. We hope to see more in the future.

AWIPS II Report – Michelle Mainelli

Michelle's Slides

Highlights Include:

64-bit news:

Updated migration schedule:

GEMPAK news:

NCEP still notes performance issues with model data. These are being worked on and should be resolved before release in spring 2013.

NCEP will be producing webinars; these can be made available to University community. Raytheon has provided admin documentation (1400+ pages), Michael has been distilling into a smaller set of admin instructions for Unidata community.

Michelle also described some "extended" projects, including thin-client software and collaboration tools.

AWIPS II Report continued – Michael James

Michael's Slides

Michael discussed options for Unidata release of AWIPS II, including timing and options for what to include in the release. There are concerns about releasing a package that is not completely functional, but Scott Jacobs (NCEP, on the phone) suggested there are nuances of functionality to be considered.

Bottom line: NCP is expected to be "mostly" complete by the projected time for release to Unidata community (Spring 2013). D2D is "mostly" complete now. Even if not all GEMPAK work is done by release time, AWIPS II should be in reasonable shape for Unidata community.

Support: Mohan notes that Unidata never envisioned providing support for anything other than the NCP. What would it mean to release a package that includes D2D as well? What will the support costs to Unidata be?

Lunch Break

DeSouza Award

David Knight (University at Albany, SUNY) was presented with the Russell L. DeSouza awarded for outstanding community service. David's presentation was titled "Collaboration, Cooperation, and Community."

David's Slides

A video of David's talk is available on the Seminar Series page.

iPython demo – Rich Signell

Rich Signell gave a brief demonstration of the iPython notebook, which runs in a web browser and talks to a server running a Python interpreter. Great possibilities for interactive visualization through a thin client (browser). See iphython.org

Rich later pointed out this video: http://www.youtube.com/watch?v=HaS4NXxL5Qc.

Status Reports

Committee members had read the staff status reports before the meeting. There were few requests for clarification

CONDUIT – Becky Cosgrove

Becky's Slides

Becky Cosgrove (via telephone) described changes to the CONDUIT data stream. Several models have received upgrades.

The change from RUC to RAP model output is now scheduled for 1 May 2012.

While some similar products exist in both CONDUIT and NOAAPort, there is no exact duplication of products.

NCEP will be updating CONDUIT servers after move to new building; additions to the CONDUIT feed can be implemented after the upgrade.

CONDUIT user survey:

CDM-GRIB Variable Naming – John Caron

John's Slides

John described a problem in the current CDM implementation that results in problems with netCDF variable names generated from GRIB files. He also provided some context on the various problems with GRIB tables, and referred to his paper on the subject. (See also other related writings on John's home page.)

To fix the existing problem, at least some variable names must change. (That is, when the next release of the CDM reads a GRIB file, one or more variable names created from the data in the GRIB file may be different than they were with the previous CDM release.) This will cause some IDV bundles to break, and also cause problems with existing TDS URL's, as well as other potential issues.

All solutions to the problem involve breaking something. John outlined some possible courses of action:

  1. Change the minimum subset of variable names, leaving the others as they are. With this approach an estimated 20% of variable names will change with the next CDM release. Some hand-coding might minimize the disruption to bundles, but there will be some pain. In the future, names will continue to "break" as GRIB tables slowly evolve.
  2. Institute a wholly new naming scheme that changes all the variable names once, then keeps them stable into the future. This approach creates variable names that are not particularly human-friendly, but ensures that bundles etc. do not break in the future as GRIB tables change. Users would be asked to accept a lot of pain now for the promise of less pain in the future.

Committee opinion was strong that (1) human-friendly variable names are desirable, and (2) a small amount of ongoing pain from name changes is "the nature of the web," and should be expected. Committee agreed to think the issue over and revisit tomorrow.

Workshop Logistics

Tuesday, 3 April 2012

Workshop Logistics, continued

Transportation for Wednesday is an issue, so dinner will be on Thursday evening.
Bus transportation still needs to be confirmed between hotels and Mesa lab each day, including after the reception on Monday and the dinner on Thursday. Ginger Emery is working on this issue. A hike on the Walter Orr Roberts Trail will be planned during one of the breaks.

The reception on Monday late afternoon will provide an opportunity for participants to display and present posters. The students are encouraged to create posters, in particular, but any participant can present a poster. The registration asks if the participant is going to present a poster.

The student applications will be judged by the workshop subcommittee. The due date for submission is 18 May 2012.

The question of laptops and what software is needed was discussed. The speakers need to be contacted to obtain this information.

The planning committee took a field trip to the Mesa Lab. Tina Campbell showed them the rooms available for the workshop, and they decided that the Chapman room is too small, so that one would be eliminated from the program. Everybody was pleased with the Damon Room and the Main Seminar Room. The Damon will work out fine for the reception/poster session, and the Main Seminar Room will be great for the plenary and half of the break out sessions.

Following the workshop, Jennifer Collins and Kevin Tyle will share the lead to produce the BAMS article.

The Users Committee will meet a day prior to the workshop to get things coordinated and organized.

Blue Sky Session

Machine specs for AWIPS II-64  bit-can use 64 bit for workstation, both for AWIPS server and EDEX

Server and CAVE clients

Question of use of NAWIPS -everything!  PGEN, NMAP II, NSHARP-

Python lets you use so many things-Python notebook  netcdf 4
Server side processing-use python.
Bridging connection between Jython and Python. 

IDV in classroom-add capability of 2 soundings on one-temp invection and isobars; create lines to be able to compare layers 

Develop an ability within IDV to read non-height vertical coordinate data, such as model output redistributed in isentropic coordinates.

Skew-Ts in IDV- isobars, plotting two Skew-Ts on top of each other (like 0Z and 12Z), the ability of adding a box or line on top of the plotted skew-t...

data needs to be on a server

ability to plot multiple fields

More Jython, more emphasis on scripting for IDV
time-height probe

netCDF-several conventions (netCDF for WRF would be useful)

Training- WebEx classes for AWIPS II-need to record the sessions for review later

Gembud theme for a few sites who are early adopters - they in turn train the next wave

Weather lab taken away for awhile-IDV basics, hurricane models, wants to get the weather lab back to move forward

Re-focus IDV effort on the Collaboration tool (still available and functional as a plugin), so as to facilitate interactive research/operational map discussions. Hope to explore this tool beginning this fall between UWisc-Madison and UAlbany, and provide feedback to the IDV developers.

Develop an ability within IDV to read non-height vertical coordinate data, such as model output redistributed in isentropic coordinates.

Cloud computing-HTML5-converging on a standard flash, GPU's 3-D , world is full of
exciting opportunities using the cloud

Thin client-Java script competitions-keep doors and minds open in the cloud world.

Question of moving Java based desktop to HTML5
How much work is done on the client and how much on the server

AWIPS II thin client- Michelle-where is processing being done for thin client?

netCDF driver-plug in for awips2

Unidata would have to host a repository for AWIPS II - needs a lot of discussion

Server (all the work/processing done on local machine)
vs Client doing it and then pushing out the end product (e.g., image generation)

Unidata-in-a-Box – Mike Schmidt

It should be possible to create a virtual machine environment populated with Unidata software packages that people could download and run with little or no configuration.

This might be useful in environments where expertise to install/configure different operating systems is limited. Some expertise in the environments represented by the virtual machines would be required, but maybe less than needed to actually configure hardware etc.


Tom, Mike, Robb are interested in pursuing this.

ACTION: Tom, Mike and Robb should create prototype VM distribution for evaluation by committee members and others.

Workshop wrap up

Workshop to be open to undergraduates as well as graduate students

ACTION: Update workshop student announcement to include undergraduates and graduate students; add speakers to speaker list; update web pages; publicize.

CDM-GRIB variable naming wrap up

The committee recognizes the tension between making GRIB variable names human readable vs keeping them stable, in the face of changing external GRIB parameter tables. The committee also recognize that the netCDF-Java library needs to fix incorrectly named GRIB variables. With that in mind, the committee recomends that:

  1. GRIB variable names should be kept human readable.
  2. A mapping between old and new names should be created so that users don't experience disruptive changes in applications, scripts, and IDV bundles.
  3. A small, ongoing amount of "bundle breaking" is expected as GRIB parameter tables change.
  4. The IDV should develop functionality to gracefully transition bundles and other scripts as GRIB variable names, URLs, and other external web resources change.

The committee also agreed that Unidata should advocate for changes to processes used by the GRIB community (creating a registry of GRIB tables, for example), specifically by making recommendations to the WMO committee that oversees the GRIB format. Michelle Mainelli offered to have her colleague Jeff Ator, who is NCEP's representative to the IPET-DRC (Inter-Programme Expert Team on Data Representation and Codes), present Unidata's recommendations at the next team meeting in May 2012.

ACTION: John to draft a course of action for CDM 4.3 and circulate to committees.

ACTION: Ethan and John to draft recommentations for the WMO.