Unidata Users Committee Meeting Summary

18-19 April 2013
Boulder, Colorado

 

Action Items
Committee Members
Acronym List

Members Attending

Kevin Tyle, Univ of Albany, Chair
Michael Baldwin, Purdue University
Martin Baxter, Central Michigan University
Steve Lazarus, Florida Institute of Technology
Sam Ng, Metropolitan State University of Denver
Russ Schumacher, Colorado State University

Student Rep:
Stefan Cecelski, Univ of Maryland

NCEP Reps:
Becky Cosgrove (CONDUIT) (Remote)
Michelle Mainelli (GEMPAK-NAWIPS/AWIPS II) (Remote)

USGS Rep:
Richard Signell (Remote)

Absent:

Anne Case Hanks, University of Louisiana Monroe
Jennifer Collins, University of South Florida
Bart Geerts, University of Wyoming

UPC Staff Attending

Sean Arms
Julien Chastang
Ethan Davis
Doug Dirks
Ben Domenico
Steve Emmerson
Ward Fisher
Dennis Heimbigner
Marcos Hermida
Yuan Ho
Michael James
Lansing Madry
Linda Miller
Terry Mitchell
Mohan Ramamurthy
Russ Rew
Mike Schmidt
Jeff Weber
Tom Yoksas

Meeting agenda

Thursday 18 April 2013

Administrative Items
Fall Users Committee Action Items
Fall Policy Committee Action Items
Director's Report - Mohan Ramamurthy

Mohan's slides

Highlights:

Mohan announced the new UCAR Community Programs (UCP) director, Emily CoBabe-Ammann. She took the helm April 15, 2013.

The UPC has had some staffing changes: Departures-Robb Kambic and Tina Campbell. Linda Miller is working at a reduced schedule. Unidata is hiring a student intern this summer.

The DeSouza Award will be presented to Larry Oolman, University of Wyoming, at the Fall Users Committee meeting. Several inches of snow hampered travel to Boulder for Larry to accept the award at this meeting.

The NSF core funding proposal will be submitted to NSF in late May. The Unidata 2018: Transforming GeoScience through Innovative Data Services proposal is a five year, approximately 25 page proposal that provides future directions, and in part, provides a roadmap developed through the strategic planning process with the governing committee's (Policy Committee and Users Committee), surveys, and staff's assistance. The five areas emphasized are: Data Services; Software and Tools; User Support and Training; Community Services; Cyberinfrastructure Leadership.

NetCDF is supported on windows platform with improved interoperability. CF-netCDF data model extension became an OGC standard 14 February 2013. Work continues to incorporate netCDF-Java into C-based libraries.

Rosetta-conversion of ASCII to netCDF for easing use in Unidata software. The project is mostly focused on ACADIS data. Some data folks just want better formatted data and don't necessarily care about netCDF, per se.

Software and support metrics are dominated by netCDF users, however IDV usage continues to rise. Since 2010, usage has doubled.

AWIPS II - Michael and Michelle will provide a presentation later. Broad release to Unidata community by the end of 2013. There was a training course last week for Unidata and COMET folks (coordinated. by Jim Poole, NWS/Kansas City Training Center and Linda Miller, Unidata). See the Photo of participants.

EarthCube - Unidata is actively involved in various EarthCube projects. Unidata organized a workshop on

"Shaping the Development of EarthCube to Enable Advances in Data Assimilation and Ensemble Prediction"

Forecasting with CSU's Russ Schumacher. The workshop was well attended.

The latest regional workshop was conducted at San Jose State University. Regional workshops serve as a good venue to use for training and assisting several sites and to bring people together to get acquainted and to reach out to help each other.

An IT survey was conducted by UPC where 33 universities responded. The summary of the survey is here.

UPC conducted a science survey where they solicited 21 universities. Summary of that survey is here.

Equipment awards solicitation was sent and - deference to those who offered to share their THREDDS AND RAMADDA servers.

Software training workshops will be conducted this summer. The announcement for registration is posted and a few registrations are trickling in already.

CONDUIT status - Becky Cosgrove (remote)

No real recent changes. Current infrastructure cannot handle additional data due to limited LDM queue size. Updates are on hold pending a re-examination of their relationship with NOAA's Web Operations Center which hosts CONDUIT.

RAP 13 km data is now on NOAAPort for all cycles. Does the Unidata community still want this on CONDUIT?

ACTION 1: Tom Yoksas to check with RAP requesters to see whether what's on NOAAPORT is sufficient.

AWIPS II UPC status- Michael James

Unidata Configuration - A single yum script installs entire EDEX server. A 'single keystroke installation' should be very helpful to users. Currently there is a huge users manual. Kevin asked about the bottlenecks associated with multi-radar feed. Michael indicated that Raytheon fixed or added an additional thread and that he's done some work with improving this. This is especially important for the national mosaic.

UPC's release to community will be based on Oct/Nov release from NWS.

Michael to create reduced set of AWIPS II docs (1400->800 pages)

Kevin Tyle mentioned errors when CAVE loads. Michael responded that all current efforts have been on resolving data flow issues, not CAVE yet.

AWIPS II NCEP status (pdf) AWIPS II NCEP status - Michelle Mainelli and Dave Plummer (remote)

National deployment Q4FY15 - Unfortunately another delay...
8 WFOs have AWIPS II operational. This generated a bunch of Discrepancy Reports (DRs); they stopped deployment to address those and will restart deployment in Q3
Performance issues in NCP were solved, but new ones appearing with additional data ingest. NCP code will be baselined by late August, and included in the OB13.6.1 release in November

Functionality to come:

Gempak programs to read from both AWIPS II database (through AWIPS II services) and .gem files. Dave Plummer thinks this should be done in August baseline code delivered to Raytheon. NAWIPS/GEMPAK updates 6.9 (April 2013) and 6.10 (July 2013). These updates will include specific requirements for fire weather, SPC, and NHC.

Ethan asked about OpenDap access. AWIPS II will receive an OpenDap-related data delivery in May. No plans for OpenDap capability for N-AWIPS/GEMPAK.

Radar Mosaic, satellite, solar, ASCAT, Deterministic and Ensemble model data, conventional obs

Kevin asked about the loading of gridded model data. They have a test case where it took ~10 secs to load 20 panels.

The Thin Client is AWIPS-II CAVE running in remote mode connected to an instance of EDEX via an Apache proxy server to access data. Several questions have popped up - Tom Y asked whether the EDEX server can sustain multiple clients at once. Michelle couldn't answer it. Ethan asked why an apache proxy server was needed. Apparently this is a firewall issue. Thin client runs on Linux or windows with appropriate graphics card. Unclear what the max # of QPID connections an EDEX server can support.

ACTION 2: Michelle will ask Steve Schotz the following question:

Mohan asked about using it in the field on an IPad .... Michelle indicated that this was off the radar at the moment.... But might be pursued later.

Data Delivery - NOMADS came up. Becky Cosgrove is the point of contact for NCEP NOMADS issues. Michelle talked about "data discovery" requests.

NCEP baseline funcationality should be in 13.6.1 release in early November 2013.

2014 development? Space weather, high seas, N-Flow, Hazard Mapping, time series

Unidata is a critical partner - weekly telecons take place.

Mohan - is anyone working on putting AWIPS on the cloud?  Answer - no.

Ethan asked about OpenDap again. What software is EDEX using to access OpenDap and grab data? Apparently this is not clear and Michael asked for more documentation.

Users Committee Check-in

Kevin Tyle:
Keeping pace with NAWIPS releases. Recent Gempak fix allowed gdpvs to work again. Some students starting to use IDL again, MATLAB in heavy use, as is NCL. A bit of a problem to support all the different packages, keep groups talking to each other in terms of data they are using. 5th year of computer applications class for Juniors; spent time pushing IDV, creating /posting bundles. Dig down to ATM350 class folder on Albany RAMADDA server. Students will be making a final class presentation using IDV and N-AWIPS to generate their graphics.

Sam Ng:
Spend first weeks of weather lab showing GEMPAK/IDV. Students frustrated by slow load when getting data from motherlode. Students generate their own case studies. We have local repository of past month. Investigate putting best of student case studies on motherlode, or get help setting up a RAMADDA at metro state.

Marty Baxter:
Finally gave up on GARP. All synoptic case studies from GARP now in IDV; lots of work but students were pleased. Similar setup to Albany. Not using the LDM feed at all. (Organization of RAMADDA?) Took intermediate python course (with Julian). Getting an electronic map wall, will put up GEMPAK/IDV products.

Stefan Cecelski:
Still using NCL and IDV to do realtime imagery. Looking at google earth output/kml. UMD is quite disconnected from Unidata. Students like realtime data from Unidata but faculty not as enthusiastic.

Steven Lazarus:
Has become an IDV convert, use it exclusively in 1 credit weather discussion class. Create exam directly out of IDV graphics. Using LDM feed to run flat panel display (running NMAP2).

Mike Baldwin:
Our situation is similar to Steve's; gone from having our own physical server in the building, with staffing, to having a vm on a college server farm. No one in campus IT understands how to run LDM, decoders etc. Had students using IDV, nmap2; left it up to students what to use, and they mostly chose nmap2. A lot of frustration with maintaining the technology and data streams. Started project with state DOT to help with winter weather decisions; needed to be able to share info. Ended up going with google docs -- easy to use but can't do science with it. Need good way to share info with decision makers. (MR: look at google open layers) Moving towards python to visualize WRF output.

Russ Schumacher:
Idea from December workshop to develop distributed national ensemble. One of the issues we'll run into is sharing the data that we're going to generate if ensemble takes off. Got an electronic map wall up and running, still in progress. Scaling up use of LDM for common data sets, creating an archive.

Status Reports

Committee members had read the staff status reports before the meeting. There were a few points of discussion:

Equip Awards - went well yesterday - winners will be announced in the Unidata web site news

AWIPS II / GEMPAK - Marty asked about Gempak 7.0...was worried it wouldn't work without EDEX server. Not clear.

Unidata citation site (software, products, etc.) - Should draw peoples attention to the citation.

LDM - Idea for LDM 7.0 came from a prof. at Virginia. Used an EGER grant funded a neat project to reduce the bandwidth used at places like Unidata.

McIDAS - McIdas V parallels IDV - but is behind latest IDV.

Rosetta - Create and advocate for standard specs on taking pt samples. There's no set structure to ASCII files. Excel compatible. Driven by ACADIS user community.  NetCDF to Excel and vice versa. Need volunteers/beta testers.

ACTION 3: Committee members with obs data sources/sensor data amenable to importing into Rosetta should contact Sean to serve as usability testers. (Mike Baldwin, Steven Lazarus offered)

Rich Signell on the line. Validation services for Discrete Sampling Geometries? Check for certain file attributes? Suggested that Rosetta do something like this...otherwise not sure you have the conversion right.

RAMADDA alternatives - Ethan Davis

Ethan's slides

Systems researched:

Ethan - RAMADDA Alternatives? ScienceBase (USGS) interested in connecting with TDS. Rich Signell notes that it's worth making a trip to Fort Collins (USGS) to talk with them because they started with the RAMADDA functionality as their baseline. Maybe EarthCube proposal?

All researched systems support data uploading, adding metadata, discovery, http download, restful web API. Some of them support more advanced options. None support server side processing. So no pulling up a time series, for example, as can be done with RAMADDA. NcETL and Science Gateway Framework have interest in collaborating. Recommendation is to stay in contact with them; no resources at present to do anything further.

Rich pointed to http://geo.gov.ckan.org/dataset as an example.

Jeff Weber pointed out that RAMADDA is being used by NASA and NOAA (Peter Griffith/ABoVE project)

Last fall's joint meeting recommendation for Yuan Ho to devote partial time to work with RAMADDA remains in place; due to demands of getting IDV 4 released, this has not been accomplished yet.

Proposal Status - Mohan Ramamurthy

Mohan gave a summary of the NSF proposal for the next 5 years. Shared an Outline and then followed with details.

Uncertain Future!  Funding for IT has decreased in the University environment. Cloud computing has matured! It is being embraced by academia.  Even some universities have their own cloud services. Not just email and course ware- this also includes science! Matlab, etc. Gave an example - Cornell. They are providing tools! George Washington U. as well.

BYOD - Bring Your Own Device. University computer labs are being shut down. Tablets are outselling laptops, laptops are outselling PCs. Students have grown up in a world without dumb terminals, telnet, FTP. They want simple tools - browsers and apps! Should we be looking to phase out old stuff - bring in the new?  Need to be 'bold but not crazy'. Adaptation to change is important.

Comments:

Survey Discussion

There was a short discussion of the two surveys conducted by the UPC since the last meeting: the IT survey and the university science survey. Did not go over the results, but there were a couple of comments:

Russ Schumacher: As we work to make tools more transparent and easy to use, but you may greatly reduce the pool of people who actually know how this stuff works.

Linda Miller: People are suffering from survey fatigue -- why did so few people respond?

Mike Baldwin: I probably get at least one request like that per week -- my first reaction is to delete.

Break for dinner

Friday, 19 April 2013

HRRR Data

Jeff Weber reports:

HRRR data has been coming in since March. We'll be getting full 3D fields as of today. We are now getting the Sigma levels; Should we have pressure levels instead? Need feedback from usercom.

Data are fairly big: 8-10 Gb/run

Currently testing: not bringing in to the IDD cluster yet. Long term objective is not to push -- want to provide pull access via TDS. The longer term goal to create a script that lets you get required fields for WRF init.

Mike Baldwin: Don't know if the preprocessing is set up to use the sigma levels

Russ Schumacher: Don't know if WRF can deal with sigma levels -- want pressure

ACTION 4: Jeff Weber will change to ingest pressure levels rather than sigma

Jeff Weber: GSD is providing updated GRIB tables. We are willing to push the data from gale (one of UPC's machines) hourly.

Kevin Tyle: We've been pulling via LDM from GSD (Pressure levels). 2D fields only; a cool thing is that time step is every 15 minutes.

Marty Baxter: Would it be possible to have just the surface fields on Motherlode?

Kevin Tyle: We're getting just 2D, output every 15 minutes

Russ Schumacher: 2D surface fields would be good; Mike Baldwin agreed.

Jeff Weber: FIM is not coming in yet (Global model). Navgen is currently available on TDS

CMU IDV/RAMADDA Case Study - Marty Baxter

Marty's slides.

Marty gave a nice talk on creating six case studies and the use of IDV and RAMADDA. This is a junior level class, using case study events from COMET archive. Most files are in GEMPAK format. Mary puts together bundles to help student answer questions.

Support requests - list of problems/keeps track of them 6 case studies!

CMU has its own RAMADDA. Marty has developed bundles and some nice exercises to go along. Group projects. Currently saving exercises in the cloud (e.g., Google Docs) - could save in the RAMADDA. But Marty said he was unable to get the RAMADDA to hide the posted work from the student teams.

Students are happy with his assignments. Issue with color table (building one's own). Cross-sections > Map and Transect view.  How do we use the IDV most effectively for education? Marty said he liked working with IDV tabs - multiple displays - open at once. Easy to bounce around. Having the data local a huge difference - quick load. Memory upgrade from Isentropic surfaces...built first in Gempak. Don't have to do this - can do the interpolation within IDV. Subtracted out storm motion in IDV - for storm relative flow.

Data Challenges

Radar products - composites? Can do hydrometeor type. Tom Y said... Group liked this idea. With dual-pol there are lot's of possibilities.

RAMADDA is good since easy for students and faculty. Marty said that learning ADDE has been tough and needs help from staff. Tom Y is willing to remotely log into any site that needs hep setting up ADDE. Plans to show this to Marty after close of meeting. Will also assist Kevin at U Albany.

IDV issues:

Can CSU begin to archive radar composites (N0Q mosaic with regular structure (15 min?) for archive) Maybe use an ADDE server?


ACTION 5: Tom Yoksas will check on having both N0R and N0Q at once.

Can IDV have a regular release schedule July and December? Fits with academic schedules...

Should we have a formal meeting to share "what you're doing with IDV"?

Site visits might be less intimidating than regional workshop

Idea: Create a way to share desktop with support -- make a support appointment? WebEx / GoToMeeting?

BlueSky Session

Sam Ng
RE BYOD: make an app or lite version with model data, don't need all the models. Augment radar and satellite apps that exist from elsewhere (U. Wisc.). In IDV- still can't get upper air data to work in IDV.

Mike Baldwin

Russ Schumacher -
NWS is heading towards AWIPS II. COMET is losing its NWS support however. Who will do Training? How is all of this going to work? How does our focus on NWS related activities like the AWIPS2 migration and the EDEX server issues factor in to the kinds of effort from Unidata and the community at large?
University ensemble issue: will there be a proposal to do this as a prototype? How do we make the data available, who will store, etc. Issue of client server calculations. Post-processing issues.  Data needs to be in one place.

Steven Lazarus

Stefan Cecelski
How feasible would it be to make the IDV use a google map as a default background? What is the status of ISL scripting; can it be enhanced?

Rich Signell
Enthusiasm for R & Python is building -- can a common data model library fit into these environments? MET office has package called iris (google scitools iris) all on github. Richard Hattersley is tech lead. Look for video of Hattersley describing how they chose this direction. Rich will circulate links.

Kevin Tyle
Visited Dave Dempsey in San Francisco. Is there a way we can facilitate sharing the products we create?
Can we reach out to students with a monthly IDV bundle contest?

How about a Usercomm blog?

ACTION 6: Kevin will follow up with UserComm members to carry out site contacts over the summer before the fall meeting.

New Datasets - Mohan Ramamurthy and Linda Miller

Does community want access to IRIS surface pressure data? Already available via web. Should it be distributed via the IDD? Potentially interesting. Mostly eastern US.  The IRIS folks are seismologists. See http://www.iris.edu/dms/products. Mohan tasked committee to look at the data and provide feedback.

LAPS has been around for 25 years, they are looking at what to do going forward. Unidata doesn't have any experience with LAPS, but would be very interested in integrating output from LAPS into AWIPS II. Perhaps we should wait until an AWIPS II perspective is available, rather than distributing/supporting just the model. Don't want to get into the business of supporting/distributing a model.

Brent Gordon is eager to distribute Space Weather data. The volume is not large.
Mike Baldwin: We absolutely want this.

ACTION 7: Unidata (Linda Miller) should pursue gaining access to space weather data.

A special thanks to Steven Lazarus for providing his notes from the meeting!

Linda Miller and Doug Dirks
Community Services - Unidata
University Corporation for Atmospheric Research
P.O. Box 3000
Boulder, CO 80307-3000
303 497-8646 fax: 303 497-8690