by Murray and McWhirter
Sunday, 1/29/06, 5:30 PM
Paper P1.14 in 15EDUCATION1
Unidata has been developing the Integrated Data Viewer (IDV) as a general purpose tool for displaying and analyzing geoscientific datasets. The freely available, Java™-based reference application is easily installed on most computer platforms. While the reference application's interface is geared toward atmospheric science, the underlying framework is discipline agnostic. Data only needs to be georeferenced to be displayed in the IDV. Because of this, the IDV is being used in oceanographic, hydrologic and geophysical disciplines, as well as atmospheric science. Some users are accessing and displaying their data in the reference application while others are creating customized versions that meet the needs of their users.
The latest (1.2) release supports more formats of data through the netCDF for Java's Common Data Model (CDM) interface. Support for direct reading of GRIB, GRIB2, WSR-88D Level II & III, point observation and aircraft trajectory data from local disk and remote servers has been added. Model data on staggered grids from WRF and ROMS models is supported through this interface. The underlying VisAD data model enables the IDV to easily combine multidisciplinary data from local and remote sources in the same interface. The IDV supports several GIS data including shapefiles, GeoTIFF and data from WMS servers. All these features make it an excellent tool for geoscience exploration.
by Yee, Williams, Blackburn, Carson, and Craig
Monday, 1/30/06, 2:30 PM
Paper P1.2 in 22IIPS1
In the summer of 2005, an operational demonstration of the newly-developed NCAR Turbulence Detection Algorithm (NTDA) was performed by NCAR's Research Applications Laboratory under direction and funding from the FAA's Aviation Weather Research Program. Real-time NEXRAD Level II data from 16 radars over the upper Midwest were obtained from the National Weather Service over Internet 2 and ingested via Unidata's Local Data Manager (LDM). These data were processed by the NTDA, a fuzzy-logic algorithm that computes in-cloud estimates of eddy dissipation rate (EDR). These data were then combined to produce a three dimensional turbulence detection mosaic every 5 minutes. The mosaic was made available for viewing via an interactive Java web-based application. In addition, aircraft route and position information was ingested via the Aircraft Situation Display to Industry (ASDI) data stream and used along with the turbulence mosaic to produce text messages depicting turbulence over a 60 nm by 120 nm region ahead of selected aircraft. These customized messages were in turn uplinked via ARINC to ACARS printers in the targeted United Airlines aircraft cockpits. A website was created to allow pilots to review the series of uplinked messages and provide feedback via a questionnaire and comment form. This paper describes the design and outlines some of the challenges encountered in the implementation of the Turbulence Remote Sensing operational demonstration system.
by Wang, Davies, Huang, Olson, Otkin, Yang, Wei, Niu, and Turner
Monday, 1/30/06, 2:30 PM
Paper P1.28 in 14SATMET1
A fast and accurate hyperspectral infrared clear/cloudy radiative transfer model is developed to simulate the Top-Of-Atmosphere (TOA) radiances and brightness temperatures over a broad spectral band (~3-100µm). The principal use of this effort is to generate TOA brightness temperature, radiances, and surface to space transmittance over large spatial domains for realistic surface states and atmospheric conditions to assist in retrieval algorithm development for next-generation hyperspectral IR sensors.
Cloud is a critical factor in radiative transfer model. Most of the models deal with one layer cloud only. Observations both from space and in situ indicate that multiple layer clouds occur more than 50% time globaly, most of them are two layer clouds. A two-layer cloud model in the framework of the GIFTS fast model (LY2G) has been implemented and an ecosystem surface emissivity model (MODIS band resolution) has been included. An automatic selection of cloud layer type, top height, optical depth (OD), effective diameter (De) from mesoscale model outputs, and/or from the atmosphere profiles with a two-layer cloud formation model has been developed and incorporated into LY2G. Overall, this model (LY2G) runs less than 1s per GIFTS spectrum (3000+ channels), and yields more accurate results in comparison to one layer cloud model in reference to a complete and sophisticated radiative model, i.e. LBLDIS.
For the purpose of comparison and validation, a range of simulations were performed for generating LY2G and LBLRTM/DISORT simulated brightness temperatures for GIFTS channels and equivalent cloudy profiles. Results show that adopting two-layer cloud model at least doubles accuracy of simulations in comparison to one layer-cloud model, which will eventually increase retrieval algorithm accuracy. A netCDF interface option was added to make easier the visualization of inputs/outputs with Unidata's IDV.
Monday, 1/30/06, 4:00 PM
Paper 4.5 in 22IIPS4
The IDD-Brasil is the result of cooperation among Brazil's Centro de Previsão de Tempo e Estudos Climáticos (CPTEC, a division of INPE), Brazilian Universities like the Universidade Federal do Rio de Janeiro (UFRJ), and the US Unidata Program Center. It is the expansion of the Unidata Internet Data Distribution (IDD) system in Brazil, and now is delivering the full set of global observations, global model output transmitted distributed the US NWS in NOAAPORT, and all GOES imager channels in near real-time to top level redistribution nodes established at CPTEC and the UFRJ. From there, the data is being relayed to a rapidly increasing community of university users
At CPTEC the data sharing component of the IDD-Brazil is being used to disseminate the output of the CPTEC models, satellite imagery, satellite derived products, and hard-to-obtain mesonet and automated reporting network observations to university participants in both the South American IDD-Brazil and North American IDD. Dissemination of the observational data is important given the sparse coverage of WMO synoptic reporting stations in Brazil. The numericals outputs from CPTECxs regional models have the highest resolution available for South America, like the operational ETA in 40 Km scale grid, and the pre-operational ETA in 20 Km scale grid. GOES satellite imagery also is operationally collected at CPTEC, and specific products are being implemented to be ingested on IDD, like high-resolution sectors of South America. Soon several others additional CPTECxs products for south america area will be inserted on the system.
All these data ingesting on IDD system makes high-resolution specific products for South America easily and freely available for universities and meteorological centers. Although these data are freely available on CPTECxs internet page, they arenxt available on the WMOxs GTS system. It is envisioned that data sharing efforts such as these will foster new collaborations among Brazilian meteorological centers and universities and their counterparts throughout South, North, and Central America.
by Mohan K. Ramamurthy
Tuesday, 1/31/06, 8:30 AM
Paper 6.1 in 22IIPS6
A revolution is underway in the role played by cyberinfrastructure and data services in the conduct of research and education. We live in an era of an unprecedented data volume from diverse sources, multidisciplinary analysis and synthesis, and active, learner-centered education emphasis. For example, modern remote-sensing systems like hyperspectral satellite instruments and rapid scan, phased-array radars are capable of generating terabytes of data each day. Complex environmental problems such as global change and water cycle transcend disciplinary and geographic boundaries, and their solution requires integrated earth system science approaches. Contemporary education strategies recommend adopting an Earth system science approach for teaching the geosciences, employing new pedagogical techniques such as enquiry-based learning and hands-on activities. In essence, today's education and research enterprise depends heavily on robust, flexible and scalable cyberinfrastructure, especially on the ready availability of quality data and appropriate tools to process, manage, analyze, integrate, and visualize those data.
Fortuitously, rapid advances in computing, communication and information technologies have also revolutionized the use of data, tools and services in education and research. The explosive growth in the use of the Internet in education and research, largely due to the advent of the World Wide Web, is by now well documented. On the other hand, how other technological, social and cultural trends have shaped the use of data services is less well understood. For example, the computing industry is converging on an approach called Web services that enables a standard and yet revolutionary way of building applications and methods to connect and exchange information over the Web. This new approach, based on XML – a widely accepted format for exchanging data and corresponding semantics over the Internet - enables applications, computer systems, and information processes to work together in a fundamentally different way. Likewise, the advent of digital libraries, grid computing platforms, interoperable frameworks, standards and protocols, open-source software, and community atmospheric models have been important drivers in shaping the use of a new generation of end-to-end cyberinfrastructure for solving some of the most challenging scientific and educational problems.
This presentation will provide an overview of these issues and discuss the how these developments are enabling new approaches to applying data services for solving geoscientific problems.
by Yoksas, Emmerson, Chiswell, Schmidt, and Stokes
Tuesday, 1/31/06, 9:15 AM
Paper 6.4 in 22IIPS6
The mission of the Unidata program of the University Corporation for Atmospheric Research (UCAR) is to provide universities with innovative applications of current computing and networking technologies to access and use atmospheric and related data for education and research. One application, the Internet Data Distribution (IDD) system, is an event-driven network of cooperating Unidata Local Data Manager (LDM) servers that distributes discipline-neutral data products in near real-time over wide-area networks.
The IDD was developed in the early 1990s in response to challenges related to weather-data ingest via satellite broadcast (e.g., local sources of terrestrial interference, data outages caused by solar occultations, weather-related outages due to signal degradation, and the difficulty in locating satellite reception systems near departmental resources) and to provide access to datasets that were not commonly available. The IDD has been the primary meteorological data delivery vehicle used by US universities with atmospheric science curricula for over a decade. Starting with a modest goal of internet delivery of data available in the NWS Family of Services satellite broadcast, the IDD has grown to become the leading Internet2 advanced-application, currently delivering over 20 terabytes (TB) of data per week to participating institutions. Stress testing conducted at the Unidata Program Center offices in the summer of 2005 demonstrated that a cluster approach to data relay was limited only by the bandwidth available in the underlying (gigabit) network thus ensuring future IDD extensibility.
The Unidata IDD has recently expanded from a US-centric delivery system to one that includes 13 countries on 5 continents. Additionally, the LDM is being used as the data distribution engine in systems akin to the Unidata IDD: by private industry; by several US government agencies including the National Weather Service and NASA; and by the national weather services of South Korea and Spain.
This paper reviews the evolution of the LDM/IDD over the past decade, and provides a vision for future development.
by Miller, Chiswell, Emmerson, Weber, and Yoksas
Tuesday, 1/31/06, 9:30 AM
Paper 6.5A in 22IIPS6
Serving a diverse community of users, the Unidata Program Center (UPC) seeks collaborative opportunities to provide data, tools and software for education and research in earth system science. Two projects that began as grassroots efforts with the U. S. Weather Research Program, including several collaborating institutions are highlighted in this paper. At the core of both projects is Unidata's Local Data Manager (LDM) technology which delivers over 20.5 TB of data per week over Internet2, via the Internet Data Distribution (IDD) system. Examples of how CONDUIT and Level II data are being used, along with the volumes of data distributed and the overall structure that makes it work using the LDM-IDD will be discussed.
Meeting the challenges presented by CONDUIT (Cooperative Opportunity for NCEP Data Using IDD Technology), the LDM-IDD is distributing high-resolution model data sets not currently available through NOAAPORT. Through the cooperation of NOAA's National Weather Service (NWS) and National Centers for Environmental Prediction (NCEP), and several institutions willing to be test sites initially, the data are now flowing to 71 sites at 46 unique domains.
The WSR-88D Level II data distribution began as the Collaborative Radar Acquisition Field Test (CRAFT) with several collaborators (University of Oklahoma, NSSL, University of Washington, et al) interested in accessing and using Level II data from the radars. Through stakeholder cooperation, a combination of leveraging technology and creative partners forming useful collaborations, the technology was transferred to the National Weather Service in 2004, and the data are now available to the broad community of users.
by Rew, Hartnett, and Caron
Tuesday, 1/31/06, 11:00 AM
Paper 6.6 in 22IIPS6
The netCDF data model, data access libraries, and machine independent format are widely used in the creation, access, and sharing of data in the geosciences. HDF5 software, originally developed at NCSA and now maintained by The HDF Group, implements another popular data model, data access libraries, and format for scientific data. Recently developers at Unidata and NCSA have developed netCDF-4, software that implements an enhanced data model and new data access interfaces built over the HDF5 storage layer and format. The resulting software provides compatibility with existing netCDF programs and data, more powerful data modeling abstractions, and features for use in high performance computing such as parallel I/O.
We discuss how the new features are intended to be used, and make recommendations for both data providers and developers who may be considering the use of netCDF-4 for future archives or applications.
by Lazzara, Langbauer, Manning, Redinger, Seefeldt, Vehorn, and Yoksas
Tuesday, 1/31/06, 12:00PM
Paper 6.10 in 22IIPS6
The Antarctic meteorological community is constantly faced with the challenges of acquiring and distributing meteorological data. Internet connectivity to and from the Antarctic continent is costly and generally of low bandwidth. The distributed nature of meteorological research and observational efforts in the Antarctic (i.e., many small projects and local programs) results in a variety of small but valuable datasets, yet distributing these data to other researchers, forecasters and logistic decision makers has been a continuous challenge. Solutions to some of these difficulties, such as better Internet communications, are some years away. In the meantime, sharing and distributing Antarctic data has often been done in an informal and ad hoc manner, often based on personal contacts. Operational forecasting for logistical activities in the Antarctic, and active Antarctic meteorological research programs are in need of a reliable, steady flow of meteorological observations, model output, and other related data in what is a highly collaborative environment.
Over the last few years, discussions spear-headed by the National Science Foundation (NSF), the agency that oversees the United States Antarctic Program (USAP), have led to a community discussion on collaborative efforts. The June 2004 Antarctic Automatic Weather Station – Antarctic Meteorological Research Center – Antarctic Mesoscale Prediction System (AAWS-AMRC-AMPS) joint annual meetings included a discussion on the synergy of the Antarctic meteorological community. The initiative to come from this discussion brings to life a community effort, with the leveraging help of another NSF funded project, the Unidata program. The result is the establishment of the Antarctic Internet Data Distribution (Antarctic-IDD) System, the objective of which is to be a system to reliably share and distribute Antarctic meteorological data.
The Antarctic-IDD is based on the Local Data Manager (LDM) software developed by Unidata and compatible with the overall Unidata IDD/LDM architecture. Each participating site runs LDM software, which is in almost continual communication with one or (preferably) more other sites. Data files properly identified and inserted at one site into a local database file (called a “product queue”) are then available for almost immediate transfer to the product queues at other sites. The result is a collaborative network of sites, each sharing the datasets available to them. This system was setup in a test mode and demonstrated in the spring of 2005. Currently, the Antarctic-IDD is growing to include a variety of data sets from a variety of data providers for a variety of users. At this time, the Antarctic-IDD carries surface and upper air observations, satellite observations and products, as well as numerical model output.
This presentation will review the initial setup of this system, and the current status as well as outline future plans. The presentation will also touch on some issues facing the Antarctic-IDD, including data formats, data compression, firewall limitations, transmission file naming protocol, etc.
by Domenico, Nativi, Caron, Bigagli, and Davis
Tuesday, 1/31/06, 1:45 PM
Paper 8.1 in 22IIPS8
Teams at the Unidata Program Center and University of Florence are working with a number of international partners to implement a web services interface to traditional atmospheric and oceanographic datasets currently stored in netCDF form or served via the OPeNDAP protocol . The project will result in a gateway service using Web Coverage Service (WCS) specification of the Open Geospatial Consortium (OGC). Underneath the WCS interface will be a combination of technologies including THREDDS (THematic Real-time Environmental Distributed Data Services) and HDF5 (Heirarchical Data Format) in addition to netCDF and OPeNDAP. A key component of the project is to develop mechanisms for explicit encoding of coordinate system information in the form of Coordinate System extensions to NcML (the netCDF Markup Language), directly in the data files themselves and in the form of GML (Geography Markup Language) extensions to NcML. These extensions, called NcML-GML, include a subset profile of the standard GML which is in the late stages of adoption by the International Standards Organization (ISO). The WCS interface specification will be developed in the context of an OGC Interoperability Experiment called GALEON (Geo-interface to Atmosphere, Land, Earth, Ocean NetCDF). The paper presents the current status and updated objectives of the project.
by Yalda, Clark, and Joseph
Tuesday, 1/31/06, 1:45 PM
Paper J4.1 in 15EDUCATION4
As collaborative partners in Linked Environments for Atmospheric Discovery (LEAD), Millersville University and Howard University, along with several other education testbeds, are responsible for the evaluation and assessment of LEAD prototypes, and the development and dissemination of educational materials and services to the wider education community. Toward this effort, Millersville University is collaborating with other LEAD partner institutions, including local high schools in extending several existing tools such as Unidata's Integrated Data Viewer (IDV) and NASA JPL's SWEET ontology for LEAD educational initiatives. Undergraduate students at Millersville have been involved in developing interactive modules and other learning materials around the IDV. Undergraduate students have developed an IDV beginner's tutorial that is specifically designed for pre-college teacher and student users. In addition, they have created IDV bundles that serve as a basis for the visualizations within the LEAD-To-Learn education modules. These modules allow students to interact with and visualize output from the NAM and WRF numerical models, and other data types, while learning through discovery related meteorological concepts. Undergraduate students have also been involved in a significant extension of the Semantic Web for Earth and Environmental Terminology (SWEET) ontology developed at NASA JPL to include quantities of relevance to mesoscale meteorology. Over 560 new quantities have been added, tripling the ontology vocabulary, and this number is likely to reach 1000. Another key enhancement of the Millersville effort is the addition of a glossary for the LEAD ontology. The SWEET-LEAD ontology will be wrapped as a Web Service at the University of Alabama-Huntsville, and will be accessible via the LEAD portal for query, info-mining, and resource cataloging, and is an essential component in the development of a dynamically adaptive learning environment for students and teachers. Finally, Millersville undergraduates have been working extensively with Howard University to develop educational supplemental materials for Howard's Weather Camp 2005. Undergraduate students at Millersville have developed specific IDV bundles that will be utilized to enhance the instructional material, but will also be used by Weather Camp 2005 students for research projects and further discovery. This paper will report on major education activities that are emerging from the LEAD project initiatives
by Creager
Tuesday, 1/31/06, 4:30 PM
Paper 8.10 in 22IIPS8
Verification of umerical prediction data is a time-consuming and somewhat tedious task requiring significant effort to align observation sites with model output. Automation, involving acquisition of in-situ observations, and appropriate georeferencing with regard to model data outputs is both feasible and straighforward. Use of this approach allows incorporation of the observations data into a geospatial database. Use of geospatially registered model output graphics, and geospatially registered in situ data, when incorporated into a geographic information system (GIS), allows display of both the predictive data and the temporally associated observational component. Finally, incorporation of remotely sensed data, including multispectral sensors, doppler weather radar, vertical doppler wind profilers and radiosondes allows another set of verifcation data which can be incorporated rapidly using GIS techniques. Finally, creating a web-enabled service capable of providing these data on demand, and allowing data retrieval using well-documented web-services for inter-institutional comparison.
Texas A&M University is engaged in modeling efforts using the community MM5 version 3.7 for predictions used in the Texas Air Quality Field Study (TexAQS-II). This study has high-impact areas in southeastern Texas (Houston-Galveston area) and north-central Texas (Dallas-Fort Worth). The goals of of this study include documentation of modeling skill, and precise field studies of particulate, organic and photochemical elements of airborne pollution. MM5 is run once daily, starting at 00Z, with hourly predictive output for 54 hours. Graphical output is created “on-the-fly” as each hourly output file is written. Three domains are currently employed: a 36 kilometer (km) coarse grid, an intermediate grid, 12 km, and a 4-km grid covering essentially Texas. Streaming in-situ observations are received over the intermediate and fine gridded domains from a variety of data providers, using the Unidata Local Data Manager (LDM). Texas A&M receives all available Level II radar data from the National Weather Service's (NWS) WSR-88D (Weather Surveillance Radar 88 Doppler) network. These data are cached for a period of 31 days.
The Texas Mesonet has been providing data using a web-mapping system based on on the University of Minnesota's Mapserver software, PostgreSQL, and PostGIS. We have evolved from creating shapefile-format GIS data files, to using on-the-fly data requests to the PostGIS database for maps and in-situ data. We have been utilizing a georegistered Level II mosaic image to present near-real-time radar data. Thus, we can now leverage this experience to provide map-layer data overlaid to promote near-real-time verification in a highly public, reviewable and scrutinizable manner.
This presentation will detail the methods and procedures used to present these data, and initial results of verification using these tools. A real-time, web-based demonstration will be available.
by Santek, Whittaker, Hibbard, Dengel, Parker, and Rink
Tuesday, 1/31/06, 4:45 PM
Paper 7.10 in 22IIPS7
The Man computer Interactive Data Access System (McIDAS) software was developed in the early 1970s at the University of Wisconsin-Madison to track cloud features and visualize data from the then new-generation geostationary satellites. For the past 30 years, the software has been kept current by including access to data from new instruments and by adapting to changing computing hardware and display platforms. The last major transition was during the 1990s when McIDAS was moved into the UNIX and X Window System environment and with the development and use of the Abstract Data Distribution Environment (ADDE) for data access.
New sensors being developed for future operational satellites will exceed the design of the current data structures and the visualization capabilities of the McIDAS software. Innovative techniques for visualizing and developing algorithms with these new data types are needed. The Integrated Data Viewer (IDV), a reference application that is being developed by the Unidata Program Center, demonstrates the flexibility that is needed in this evolving environment, using a modern, object-oriented design approach. The IDV is based on VisAD, a visualization library developed at the Space Science and Engineering Center (SSEC) that incorporates lessons learned from McIDAS and supports a universal numerical data model, flexible 2-D and 3-D displays, a distributed component architecture, and flexible user interaction and collaboration.
A plan has been developed to transition the current McIDAS-X software into a VisAD-based system, known as McIDAS-V. The goal of the transition is three-fold:
1. Allow the extensive library of McIDAS-X heritage code to be usable in the new environment without a need to rewrite,
2. Build on the existing capabilities of the IDV, including the overlay of meteorological data from a wide variety of sources,
3. Provide a new environment for developing algorithms and new visualizations that are required for data from future sensors.
A key component of McIDAS is access to a variety of meteorological data. For the past decade, the ADDE servers have provided efficient access to large datasets worldwide. These will continue to be used in the forthcoming McIDAS-V era. To encourage the use and development of new servers, the ADDE servers and required McIDAS-X library modules are now available as an open source package known as OpenADDE. In addition, clients other than McIDAS (such as SGT or code written in Matlab and IDL) have access to these ADDE datasets via a VisAD Java interface.
A status of the McIDAS-V effort and OpenADDE use will be presented.
Tuesday, 1/31/06, 4:45 PM
Paper 8.11 in 22IIPS8
The National Weather Service (NWS) now issues public forecasts for the nation in gridded form at 5 km spatial resolution. These forecasts comprise the National Digital Forecast Database (NDFD). The update frequency is several times per day. Forecasts include 2-meter temperature, dewpoint and relative humidity, wind, precipitation amount, cloud cover, and weather type, across the entire nation. The high-resolution NDFD is a major step forward for the NWS as previously most public forecasts focused only on areas the size of counties or zones, or for specific points of high interest such as city centers. Thus the NDFD allows users to get a forecast for a point within just a few kilometers away, no matter where they live.
The NDFD grids are made available to the public using the World Meteorological Organization (WMO) standard GRIB2 binary format. Only a few applications allow native viewing of GRIB2 format data. Many users of NWS data in the U.S. are not familiar with WMO formats and would prefer the data to be in formats that integrate more easily with Geographic Information Systems (GIS). Indeed, the emergency management community is one of the agency's primary customers and is largely already using GIS software and many of their existing data sets are already in a GIS-ready format.
This paper will explore some applications of using GIS to analyze the NDFD forecast grids. This includes an automated routine for converting the entire national grid into the UNIDATA netCDF format which is commonly used by the atmospheric science community. Through the use of recent enhancements to GIS software, netCDF data will be directly readable, along with other GIS-ready datasets. Providing GIS-ready NDFD in either netCDF or GIS “shapefile” formats will allow more advanced GIS analyses. Examples include verification comparisons of NDFD grids to observational data, short-term (3-12 hours) decisional aids for emergency managers, and increased flexibility for designing forecast graphics for broader dissemination.
by Chiswell, Domenico, and Weber
Wednesday, 2/01/06, 8:30 AM
Paper 10.1 in 22IIPS10
Real-time steering of mesoscale forecast models using objective techniques allows data assimilation and computational resources to focus on Regions of Interest (ROI) where active weather will likely occur. In developments inspired by a presentation at the 2005 AMS Annual Meeting by Kelvin Droegemeier, mechanisms for using existing real-time data systems and analysis tools to steer a local forecast model to a region where "interesting" weather would occur during the forecast period have been employed which enable the model domain to evolve over successive forecast runs while providing research and education users with products and data based on the forecast domain.
The system implemented within a week following the San Diego AMS meeting uses operational forecast model fields and an objective weighting method to select the region of interest. GEMPAK's Gaussian Weighted Filter using normal distribution of weights is used to create a 24 hour predictive field where the model domain is centered over selected maximas. The method currently uses the 24 hour accumulated precipitation field produced from forecast hours 6 through 30 of NCEP's 12km ETA (aka NAM) model, which is distributed operationally via NOAAPORT, as the predictor for the ROI function. Once the model domain is determined, Workstation ETA and WRF models are run. As a result, the system sucessfully tracked the major ice storm that moved across the Southeast US and effectively shutdown Atlanta and other cities in the area. Other appications of the system will be discussed.
by Baltzer, Chiswell, Domenico, and Ramamurthy
Wednesday, 2/01/06, 8:45 AM
Paper 10.2 in 22IIPS10
An important aspect of the LEAD project is dynamic adaptivity. In LEAD, dynamic adaptability is the notion of performing meteorological analysis and forecasting on demand in response to the weather. An algorithm for choosing a location of interest based on predicted precipitation from the NAM forecast has been devised at Unidata/UCAR (See Paper in IIPS Cyberinfrastructure session). The center Latitude and Longitude of this location is being utilized to steer both the Workstation Eta and WRF models (the latter running on the Unidata LEAD test bed) with each prediction cycle (4 times daily).
The results of these regional NWP runs are being stored on the test bed, cataloged using THREDDS and distributed via OPeNDAP. We've also generated IDV bundles that allow one to view the most recent regional runs as compared with the most recently received NAM data. This work is intended to enable the LEAD team to work toward developing these and far more sophisticated capabilities within the GRID environment in which it is being developed. This paper will also discuss ongoing work to assimilate real time data for initialization of the WRF forecast.
by Caron, Davis, Ho, and Kambic
Wednesday, 2/01/06, 9:45 AM
Paper 10.6 in 22IIPS
The THREDDS Data Server (TDS) combines THREDDS catalog services with integrated data-serving capabilities, including OPeNDAP and WCS, and automatic catalog generation. The data-serving capabilities are built on the NetCDF-Java version 2.2 library, which combines the NetCDF-3, OPeNDAP 2, and HDF5 data models, into what is called the nj22/NetCDF-4 Common Data Model. TDS is 100% Java, open source, and runs as a Tomcat web server application. This paper will detail its capabilities and implementation status.
by Lindholm, Wilson, and Baltzer
Wednesday, 2/01/06, 11:15 AM
Paper 10.10 in 22IIPS10
Linked Environments for Atmospheric Discovery (LEAD) is a multi-institutional Information Technology Research effort funded by the National Science Foundation (NSF). The goal of LEAD is to create a Grid and Web Service based framework to support mesoscale meteorology research and education.
LEAD presents unique challenges integrating large data volumes from real-time observational systems as well as those that are dynamically created during the execution of adaptive workflows. The LEAD Data Repository, which manages these data, must be able to autonomously handle storage and retrieval requests generated by the LEAD orchestration in addition to directly satisfying user requests.
This paper will outline an architecture being developed at the Unidata Program Center (UPC) to support the data storage requirements of the LEAD Data Subsystem. This architecture defines a set of simple interfaces to handle the responsibilities of a potentially complex data repository. It will support capabilities such as acquiring storage resources, moving data, generating metadata and unique IDs, cataloging, managing the data, providing a discovery mechanism (via browsing or queries), and providing transparent data access by LEAD services, users, and other applications.
A prototype implementation is being developed based on UPC technologies such as THREDDS catalogs and the Common Data Model and is being integrated with other LEAD technologies. The architecture is designed to allow developers to implement each interface using the most practical solution available as opposed to adopting a large turnkey solution. Various implementations will be produced by the LEAD project and made available as pluggable modules to benefit the community.
by Wilson, Lindholm, and Baltzer
Wednesday, 2/01/06, 11:30 AM
Paper 10.11 in 22IIPS
The Linked Environments for Atmospheric Discovery (LEAD) project, funded by the National Science Foundation (NSF), is building a cyberinfrastructure for mesoscale meteorology research and education. In LEAD users can build and execute orchestrations involving multiple data sources and the web service-based tools provided by LEAD. LEAD tools and applications include data mining algorithms such as ADaM, assimilation tools such as ADAS, and forecast models such as WRF.
A major goal of LEAD is to allow users to query, import, and manage data in the LEAD domain for purposes such as visualization, storage, and as input to LEAD applications. LEAD data can be individual files, collections of files, or streams. LEAD data sets may be very large, necessitating storage in a distributed manner or on a mass storage system. These large sizes also require support for subsetting and aggregation of data.
There are three classes of data that the LEAD data storage subsystem must handle. Personal data is data that a user has brought into their space within the LEAD Data Repository. Users may store data and other resources here, and also the orchestration will store intermediate results in a user's personal space. Public LEAD data is data that is made available to the LEAD community by cooperating data providers, such as universities interested in sharing the data that they receive or generate. External data is any other data that a LEAD user knows about and would like to access. NCDC archival data is an example of this data.
Both personal and public LEAD data are catalogued within LEAD. Via the catalogs, users can query over the metadata to discover data that meets specified time, spatial, and field requirements. Other relevant requirements pertain to metadata generation, data quality, metadata quality, access control, and handling of proprietary data.
This paper presents some of the requirements for data access, acquisition,
storage, and retrieval that are shaping the architecture of the LEAD Data Subsystem.
It also gives a high level view of that architecture through a canonical use
case, as well as its current state of development.
by Clark, Fitzgerald, Baltzer, Joseph, Ramachandran, and Chiao
Wednesday, 2/01/06, 11:45 AM
Paper 10.12 in 22IIPS
Two key elements of the Linked Environments for Atmospheric Discovery (LEAD) project are dynamic adaptability and ensemble forecasting. In LEAD, dynamic adaptability is the notion of performing meteorological analysis and forecasting on demand in response to the weather. EarlyLEAD is a proof-of-concept effort within the LEAD project to autonomically identify significant mesoscale features and run the WRF model to follow the development of these features. The purpose of EarlyLEAD is to demonstrate and evaluate developing instantiations of some LEAD technologies, and to bring a subset of LEAD capabilities into computing environments that are not likely to have authorization on the TeraGrid, even when LEAD is fully functional. In addition, by adapting tools from different sources and combining them to create a new set of products, EarlyLEAD makes extensible LEAD technologies for faculty and students to use now.
EarlyLEAD begins with the most recent NAM or WRF forecast. The output from this forecast is subjected to the Phenomena Extraction Algorithm (PEA). PEA is a general purpose algorithm to detect and extract geophysical phenomena in science datasets, which does not depend on any specific domain heuristics or target data. PEA is a specialization of the Intelligent Data Thinning (IDT) algorithm developed at the University of Alabama at Huntsville (UAH) for other applications, which employs a tree based decomposition technique to recursively divide the data into multiple sections and calculates an objective information measure for each data partition. If the objective measure for a given partition is greater than a user specified threshold, the algorithm continues dividing this partitioned data further. If the objective measure is less than the user threshold, or when the data cannot be portioned further, then the algorithm terminates that recursive path. PEA and IDT are part of the Algorithm Development and Mining (ADaM) toolset developed at UAH as a component of the LEAD Service-oriented architecture.
User-defined forecast quantities, such as the u and v components of wind velocity or omega (vertical velocity) and precipitation fields, are autonomically selected from the initial model output and subjected to the PEA. The algorithm identifies and outputs the center latitude and longitude of regions of interest (ROI), and prioritizes according to their significance. The PEA can be adjusted by the user to extract the ROI according to geographical location, pre-selected meteorological phenomena, and other user preferences. Once the locations of ROIs have been prioritized, the center latitude and longitude of the ROI with the highest significance is used as the basis for establishing a high resolution WRF model domain over that region. The WRF is initialized with the model (NAM or WRF) output, then run to produce a short-term forecast. The output from the WRF forecast is again subjected to the PEA to identify where to move the WRF domain. This sequential process of output-to-PEA-to-WRF-to-output steers the WRF domain to follow the interesting weather identified through the PEA. The PEA is being tested at Millersville and Howard Universities, and at Unidata Program Center (UPC). The three WRF products are then sent to the UPC LEAD test bed system where they are made available on the OPeNDAP server and cataloged using THREDDS. The UPC Integrated Data Viewer (IDV) is then used to compare and merge the ensemble.
by Kruger, Bradley, Krajewski, Lawrence, Smith, Baeck, Steiner, Ramamurthy, Weber, Del Greco, Murthy, and Dhutia
Wednesday, 2/01/06, 2:30-4:00 PM
Poster 10 in 22IIPS
Session 2
We are developing and implementing a scientific workflow for producing surface rainfall maps suitable for research in hydrology and hydrometeorology that include the following steps: AP/clutter detection and removal, reflectivity-rainfall transformation, and georeferencing and transformation from polar coordinates to surface coordinates such as latitude/longitude, HRAP, and others. We are currently ingesting the full resolution (Archive Level II) data from 30 radars in real time via the UCAR/Unidata Internet Data Distribution. Core metadata are extracted and managed in a relational database as the data are ingested. The real-time data are augmented with selected data from NCDC's archive. We follow a watershed-centric approach so that user can request surface rainfall estimates for a watershed(s), which may require the mosaicing of estimates from several radars, but this is handled in a transparent manner by the system. Using a web browser users can select an output coordinate system, a watershed, time period, and desired temporal resolution. The system will then produce rainfall maps using predefined or user-selected algorithms in netCDF and other formats. A web services interface allows programmatic access to the system via scripts or compiled programs from remote servers. The completed system will be able to serve up watershed-centric maps rainfall maps that span multiple years. We will make the data and services available to the community via CUAHSI, NCDC, and Unidata.