LEAD at Unidata
Year-5 of LEAD in Review: Toward a Fully Functional System for Community Deployment
Nearing the end of its five year lifetime as an NSF Large ITR grant, LEAD has pioneered a new approach for integrating complex weather data, assimilation, modeling, mining, and cyberinfrastructure systems in innovative ways to empower researchers and students with capabilities heretofore available at only a few major universities and research or operational centers around the world. The key point is that LEAD brings these capabilities – using a service-oriented architecture and other relevant technologies as the underpinning – to users with the simplicity of familiar environments such as Amazon.com and Travelocity.com. By managing the complexity of inter-operative cyber tools and providing flexibility and ease in how they can be linked, LEAD allows students and researchers to focus their time on solving the science and engineering problems at hand, providing a means for more deeply understanding the tools and techniques being applied rather than the nuances of data formats, communication protocols, and job execution environments. Containing virtually all elements of modern cyberinfrastructure – from adaptive sensors and high-performance computing and networking to huge data sets, human decision making and complex virtual organizations – LEAD functionality also has been integrated with the TeraGrid as a successful TeraGrid Science Gateway project and continues to serve as an avant-garde research system for the meteorological and computer science communities. Indeed, as described throughout this report, LEAD has been a principal application driver for helping TeraGrid identify and solve some of its most important challenges and prepare for the next generation XD environment.
outcomes into impacts is a foundational goal of LEAD, and the service-oriented
architecture being developed by LEAD for conducting its research continues to
be enhanced and hardened, now operating around the
clock to support students and researchers nationwide. Following the successful pilot project
associated with WxChallenge 2007, LEAD continued to make its resources
available to the academic community. For
example, LEAD was used in METR 4133, a senior-level Mesoscale Meteorology class
much different but equally important application of LEAD is in the development
of next-generation forecasting systems that include both ensemble and
dynamically adaptive, grid-enabled features.
Once again, LEAD was a significant component of the NOAA Hazardous
Weather Test Bed in
It is important to note that the forecasts described above were supported by substantial, dedicated resources on the NSF TeraGrid at NCSA, PSC and Indiana University, and additional financial support was provide by NOAA via a C-STAR grant to the University of Oklahoma. NCAR and NCEP also produced real time forecasts for the 2008 Experiment, but only CAPS and LEAD assimilated NEXRAD Level II radar data, produced ensembles, and conducted on-demand forecasts, automatically, in response to the weather. None of these other organizations have such capability, which is a testimony to the importance and uniqueness of LEAD and the value of the TeraGrid in dynamically adaptive applications. In fact, the 2008 Experiment placed tremendous quality of service demands on the TeraGrid, and the results were considerably improved relative to the TeraGrid’s performance in 2007.
Deploying LEAD as a National Facility for Atmospheric Science and Computer Science Research and Education
From the beginning, the LEAD vision has been to not only conduct excellent research and develop exciting and powerful technologies, but to do so in a practicable way that transforms meteorological research and education and has value to other disciplines. Without question, that vision is being realized. However, the transformation can occur only when LEAD is transitioned from a research project and made available as a persistent, stable facility upon which the community can rely. This notion was expressly stated in the original LEAD proposal, with Unidata as the envisioned home for community deployment.
The outcomes and impacts realized by LEAD during its five years an ITR grant provide a strong foundation and high level of confidence upon which to build a persistent LEAD cyberinfrastructure. Based upon considerable interest in LEAD by the atmospheric science community, as measured, for example, by the strong positive feedback following the recent workshops, LEAD seeks to develop a 5-year proposal to NSF that focuses on deployment/provisioning as well as on retaining selected components of the LEAD research enterprise, leveraging the tremendous collaborations now in place to ensure that LEAD remains at the forefront of capability. This concept has the support of the UCAR and NCAR leadership as well as the Unidata Policy Committee.
Our vision for the future is a primarily service-oriented environment of data streams, historical case study-type data sets, assimilation and modeling tools, mining and analysis engines, and visualization capabilities that are as pervasive in atmospheric, ocean and Earth science research and education as are desktop computers.
The LEAD vision also involves building upon growing education and outreach programs to extend the many LEAD resources into progressively lower grade levels and into communities for which even basic capabilities are unavailable. Because weather is experienced by every human and is an excellent motivating factor for studying science, our vision includes using LEAD to stimulate interest and broaden participation in STEM (science, technology, education, mathematics) education at the grade levels where most students choose, sometimes unwittingly, to avoid science as a career. Finally, the service-oriented approach to enabling research and education has shown so much promise that Microsoft Research is considering funding a two-year program to bring essential LEAD capabilities into the Windows Vista environment including its workflow and event communication systems. This is a strong testimony to the potential value of LEAD, that the pathway taken has been appropriate, and that the successes to date reason for initiating formal deployment and sustaining LEAD as a national facility. History has shown that the most widely used, transformative systems (e.g., DODS/OPeNDAP) require at least a decade of sustained investment and hardening before their impact is fully realized.
One-Year No-Cost Extension
In July, 2008, all nine LEAD
institutions submitted and were granted a one-year no-cost extension (NCE),
which will continue LEAD as an NSF cooperative agreement through
UPC staff will continue its role in the project and provide data, software and support for the project and valuable assistance in the testing and deployment of and end-user support for LEAD systems in the atmospheric sciences community. In addition, Unidata will 1) Maintain and upgrade the LEAD portals and grid administration software and computing cluster to improve reliability and user experience; 2) Observe, modify and re-balance the LEAD storage nodes to keep space and performance maximized; 3) Continue to work with LEAD personnel to identify and solve their technology problems. 4) Continue to maintain and operate the Access Grid node at Unidata that is principally used by LEAD.
The Unidata LEAD Test Bed Status
The Unidata LEAD test bed continues to be a primary resource of data for LEAD workflows. This includes:
Figure. Data now available, both historical (on disk) and in real time, within the LEAD and broader community data catalogs.
We are exploring how the Unidata community can benefit to an even greater extent from this important resource.
LEAD Tutorial at the Annual WRF Workshop
LEAD project, lead by Unidata’s Tom Baltzer, presented a 90-minute tutorial as
part of the annual WRF Workshop in
early June, flooding in
LEAD Beyond the ITR Phase
The LEAD PIs continue to strategize about securing funding for a continued LEAD deployment facility. (Continued LEAD CS research would be pursued under an OCI CDI initiative, though that would likely not involve Unidata.).
Several LEAD principal investigators, including Mohan Ramamurthy, are planning to visit NSF on 29-30 September 2008 to meet with NSF program officers and to discuss possible scenarios for deployment as well as continued development of various components of the LEAD cyberinfrastructure. Despite repeated attempts to engage NSF officials in a dialog to discuss avenues for possibly continuing LEAD activities beyond the ITR phase, not much progress has been made for various reasons, including the transitioning of the project to a new NSF program official. The PIs sincerely hope that the conversations that initiated last year can be reinitiated that LEAD PIs and NSF will be able to engage in a dialog about extending LEAD activities beyond the period of performance for the current award, beyond the one year no-cost extension period.
LEAD Staff Departures
The uncertainty surrounding the future of LEAD and its funding, unfortunately, has resulted in the departure of two staff members, Anne Wilson (in May 2008) and Tom Baltzer (September 2008) who were funded by LEAD. In light of the fact that LEAD will be in a no-cost extension period starting on 1 October, the UPC has decided that those positions will not be filled. The UPC is working on a transition and is distributing the work around the existing staff.