|
|
| Community Newsletter |
   |
|
Table of Contents
Unidata Bestows the First Russell L. DeSouza Awardby Sally Bates, Unidata Program Center
And the winner is Harold Edmon! On 11 January 2000 at the annual AMS meeting in Long Beach, Unidata will present the first Russell L. DeSouza Award to Harry Edmon of the University of Washington. Harry’s involvement in Unidata is long and legendary. As Harry himself likes to joke, he’s been participating since before its inception. And this is true. Unidata traces its birth to a conference held in Madison, Wisconsin in July 1983. Harry was involved in many of the planning sessions that preceded that workshop. His name appears on an appendix to a report entitled “The Unidata System for University Weather Analysis and Modeling” written under the direction of John Dutton (Pennsylvania State University) as a planning document to guide the July meeting. He was also a member of the Ad Hoc Unidata Implementation Strategy Committee, whose goals were to broadly specify and design the “UNIDATA system” and identify a strategy for implementing that system. This committee met in June and prepared a report by early July 1983. Harry received his Ph.D. in Meteorology from Purdue University in December 1977 and then moved to the University of Washington as a postdoctoral research associate from 1978—80. His interest and ability in computing, however, changed the direction of his career. “By 1983 I was the director of the departmental computer facility,” Harry explained.“Cliff Mass [at Washington] and others [at various universities] had written proposals to the National Science Foundation for creating display software, and NSF had turned the proposals down. Instead of funding lots of individual proposals, they [NSF] wanted to fund a single, more comprehensive system. That was one of the drivers behind forming the Unidata program. When Unidata started to coalesce, Cliff involved me because of my technical computing expertise.” And Unidata has been relying on that expertise every since. From the beginning, Unidata has sought assistance from a group of technical experts, first under the name of the Implementation Working Group (IWG), then under the label of the Advanced Technical Advisory Committee (ATAC). Harry’s involvement has been continuous. He was the ATAC representative to the Users Committee until 1997, when he began representing the ATAC at the Policy Committee. As a result, he has been member of every committee governing the Unidata Program—no small contribution of time and patience! Harry’s contributions extend far beyond committee work, however. Ask any member of Unidata’s technical staff what comes to mind when Harry’s name is mentioned and the unanimous response is “bug fixes!” His most memorable contribution was his report of problems with the LDM5—the first version of the software designed to receive/forward data over the Internet. “We had seen that early test versions of the LDM5 performed poorly in some cases,” Russ Rew remembers, “but we were unable to easily isolate the problem. Harry, however, helped diagnose the slowdown in a classic five-line email message I still have. It not only convinced us that it was a Local Data Manager (LDM) software problem, but also made it easy to determine where the problem originated and what needed to be done to fix it. The Internet Data Distribution works as well as it does today partly because of Harry Edmon’s work helping us test early versions, and his precise reports of anomalies he observed.”Harry was also instrumental in enabling the CRAFT project. CRAFT (for Collaborative Radar Acquisition Field Test) is a University of Oklahoma effort to access and distribute NEXRAD Level II radar data in near-real time. Harry developed the prototype for capturing and compressing the data using the LDM and feeding the data to another (non-LDM-based) system. (In this effort, he was building on his earlier contribution of writing a decoder that transformed NIDS data from WSI into the McIDAS AREA file format. This decoder is what allows Unidata users to display and manipulate NIDS data with McIDAS.) Harry’s long tenure has imbued him with a certain philosophical view of today’s hot issues. “The technology may be changing, but the arguments about it never change,” he remarks. “We always want faster computers and more data; we just can’t always agree on what this means.” Worrying About the LDMby John Merrill, Chair, Unidata Policy CommitteeSuccess is pushing the Local Data Manager (LDM) to its limits. The Internet Data Distribution (IDD), which relies on the LDM, is processing ever-increasing amounts of data, both in terms of shear volume and in terms of the number of sites being served. These increases are beginning to reveal limits in the LDM’s design. While the LDM has continued to function well far beyond the data volumes originally envisioned, there are a few clouds on the horizon that have recently become topics of discussion within Unidata’s governing committees. The Known Limits to the LDMAs David Fulker, Unidata’s director, reported to the Policy Committee last September, the Unidata staff is becoming concerned about the implications of this continued growth in the volume of data distributed by the IDD. They have identified three potential problems the growth poses for the LDM. The most immediate one may be nearing solution: the LDM’s product queues. As data products stream into an LDM, products are queued until they can be processed. As more products are added to the stream, the queues must be larger, and in some cases, they are nearing their limits. The problem is with the amount of time it takes to insert or delete a product from a large queue. In the current LDM, inserting or deleting a product requires time that depends on the number of products already in the queue, leading to slow downs when there are a large number of products. In the worst case, an LDM can fall behind the data streams and lose products when there are too many products in the queue. Since Dave’s report in September, Russ Rew has developed more efficient insertion and deletion algorithms. Unidata will be testing the new design and will incorporate it into a new release of the LDM when it’s proven. The second problem is more complicated: automatic routing. In the current IDD system, when a new site asks to join the IDD system, a Unidata staff member (generally Robb Kambic) determines where in the system the site will fit. If the site is a relay node, Robb must determine not only who will send data to the site for each data stream but who will receive data from it. A backup source must also be identified for each data stream. These decisions rest on Robb’s understanding of the functioning of the underlying Internet, since network connections between some sites are better than between others and change over time. Creating and maintaining the IDD distribution trees manually is time consuming and, as the potential number of separate data streams increases, impractical. A third problem—latency—is intimately related to the problem of automatic routing. Currently, Unidata tracks how long it takes for a data product to move between the source point (say, the National Weather Service for NOAAport, for example) to an end user. Close examination of the aggregated latency data indicates that the delays between source and endpoint are increasing. While much of this may be a result of congestion in the underlying Internet, it indicates a potential problem for Unidata. As data volumes increase, problems with congestion may only get worse, leading to longer delays or even data loss. Manually re-adjusting data distribution trees to avoid areas of congestion is simply not feasible. The Advent of NIDSInto this mix of concerns comes NEXRAD Information Dissemination Service (NIDS). The NWS is planning to begin including radar data on NOAAport. Since the NWS operates 155 radars, the resulting increase in data volume and number of data products will be significant. In addition, many Unidata sites have expressed an interest in the more voluminous Level II radar data. The University of Oklahoma, for example, has built a prototype distribution network for Level II data from eight radars of regional interest (see CRAFT homepage). The inauguration of freely available radar data leaves Unidata with two distribution alternatives. It can simply add all the data to an existing data stream or it can define separate data streams. The first solution, while immediately feasible, is undesirable since most of the time most of the data from most of the radars is uninteresting. Simply adding the data to an existing stream will add large volumes of noise. Clearly, the preferred alternative is to have a separate data stream for each radar, but there are 155 NWS radars and the LDM can only support 32 separate data streams. Designing a New LDMAs Dave explained to the Policy and Users Committees, Unidata’s goal now is to redesign the LDM and the IDD system to cope with these new data in a way that is most advantageous to Unidata sites. The programÕs goal is to redesign the LDM to allow a user to subscribe to data from only those few radars with data of interest. This will mean developing a system that monitors the underlying Internet, reroutes data (continued) automatically to avoid congestion, and that is to some extent dynamic (i.e., that allows sites to change their subscription requests based on where the weather is interesting). Toward this end, Unidata has hired Anne Wilson (see the article elsewhere in this issue) whose work at Unidata will be focused on redesigning the LDM. In the meantime, shorter term fixes to the current LDM will be released as they become available. Our contribution, as Unidata users, will be to keep abreast of new LDM releases, installing them as quickly as possible, and reporting back on any problems. This is an exciting time for the Unidata community—one that promises a quantum leap in the amount of data available for our use. Progress Report: NOAAport Transitionby Steve Chiswell, Joanne Graham, and Mike Schmidt, Unidata Program CenterIn our last newsletter we told you about Unidata’s intent to transition the community from Family of Services (FOS) data to NOAAport’s NWSTG channel. In December 1998, one month earlier than planned, we turned off our FOS feed and began feeding the Internet Data Distribution (IDD) with NOAAport data from a Space Science and Engineering Center (SSEC) system connected to a COMET/NWS ground station. This configuration was backed up by a system at the University of Wisconsin SSEC. The transition was remarkably smooth. Since then, our other partners in this endeavor, Louisiana State University (LSU), Alden Electronics, and The Weather Underground (selected by Alden to be part of the network) have gone “live” as well. There were a few changes made along the way, however. We renegotiated our contract with Alden Electronics to provide LSU with proper maintenance support of its system. This reduced the number of university-based systems from two to one. The end result, we feel, is an excellent, stable, no-cost data stream for our community. Separately, the five systems mentioned above primarily feed data to parties of their own interest (e.g., the LSU site feeds other universities, the Alden site feeds its commercial customers). Together they provide protection against data loss to our collective communities—through solar outages, network interruptions, hurricanes, and hardware downtime, you will continue to receive data. Although this system is highly reliable, it is not intended for “operational” use. Continued DevelopmentWe constantly are looking for ways to bring new data streams to our community. Currently, the IDD is injecting only one NOAAport channel (NWSTG) in the IDD; however, we have been experimenting with configuring our own ingest system with an off-the-shelf, four-port board and modestly configured PC. Our initial tests show that this system provides a very stable and reliable platform which can serve as an IDD ingestion node without additional overhead. Moreover, this configuration provides a seamless method for expanding the data being distributed via the IDD as additional products are added to the broadcast. Since many of the NOAAport products do not fit under the previous FOS classifications, we will be implementing new IDD feedtypes. We also expect that NIDS data will become freely available on NOAAport in the next year or so, and would like to make that data available to our community. By experimenting with large data volumes now, we can assure ourselves that the larger data volumes can be processed without compromising the Local Data Manager or affecting IDD performance. (See John Merrill’s article elsewhere in this issue). It may also be possible that the ready availability of NOAAport reception hardware could renew interest at university sites to receive the data streams using local ground stations. If you are interested in knowing more about this development, please feel free to contact support@unidata.ucar.edu. We hope that your transition to NOAAport went as smoothly as our own; in fact, we hope you didn’t notice a change. We will continue to keep you informed as new developments or opportunities arise. If you have any general questions or comments, please contact joanne@unidata. ucar.edu. Farewell to Glenn Davisby Dave Fulker, Unidata Program Center
We have mourned and celebrated Glenn’s life in many ways, including a memorial service held May 16th at the NCAR Mesa laboratory, and a memorial Web site (website no longer available). Beyond his prowess as a software designer, Glenn performed actively as a dancer with Frequent Flyers (a troupe that incorporates low-flying trapezes into its choreography), and he loved to fly, having gained instrument and commercial ratings. Glenn attended Reed College, where his studies focused on mathematics and physics, with time spent also on art, theater, and anthropology. Glenn joined Unidata in 1987, and he leaves us a remarkable legacy. He was instrumental in creating two of Unidata’s most important software products: the netCDF and LDM packages. NetCDF (for Network Common Data Form) supports the creation, access, and sharing of scientific data in a wide range of disciplines and contexts; the package is widely used by software developers around the world, and it has fundamentally altered the practice of scientific data management. The LDM (for Local Data Manager) software is designed for event-driven data handling, and it is the linchpin in our real-time Internet Data Distribution (IDD) system, presently employed in some 160 Unidata departments nationwide (as well as a number of government agencies). Glenn conceived the fundamental idea for IDD. I remember quite clearly Glenn’s request to depart from our software development plan so he could develop an experimental version of the LDM, capable of relaying data to other LDMs via Internet. This, of course, was a seminal event, and it set the stage for discontinuing the use of satellite broadcast, in favor of Internet-based data delivery. It also is a representative event, indicating how Glenn’s creativity—and his ability to bring ideas to practice with unbelievable speed—affected Unidata’s basic nature. We have mourned the brevity of Glenn’s life and the tragedy of his death, and we continue our struggle to deal with its finality. Our hearts go out to Glenn’s family members, who travelled to Boulder from North Carolina to enrich our memorial service at NCAR. Of course we can never replace the personal relationships that had developed between Glenn and ourselves, individually and severally, nor Glenn’s wacky sense of humor that could surface at any moment, serious or light, formal or informal; but we are starting to fill the professional gaps created by his departure. And we are hoping to pay tribute to Glenn in the most significant way possible by continuing his traditions: excellence; innovation; humor; artistry; questioning the status quo; and making a difference by the force of our intellect, our productivity, and our interaction with others. MetApps: Building for the Futureby Charles Murphy, Kean University and Russ Rew, Unidata Program CenterImagine opening your Web browser and through it selecting both an analysis tool and the data you want to analyze, or being in the field and being able to compare the data coming in from your instruments with model or climatological data held somewhere else on the Internet. Imagine being able to incorporate hydrological, atmospheric chemistry, oceanographic, and/or other geoscience data seamlessly into your display. Or imagine having your students work collaboratively on a project without your presence, or imagine teaching a distance learning course involving real-time data and Unidata applications where the students are located all over the nation or world and your only interactions with them are through the Web. Or, what? What capabilities would you really like to have at your fingertips? This is not an idle question. As we’re all acutely aware, technology is rapidly transforming how we conduct our research and teach our classes. This transformation brings with it both opportunities and challenges. We have the chance to create new ways of interacting with our students and with data. The development of the Internet and new computing technologies offer the potential of creating new approaches to teaching. Technology advancements also offer the hope of freeing us from the treadmill of new software releases and from the miseries of data-format, software, and platform incompatibilites. Today’s challenges are also clear: simply keeping up-to-date is a headache for most of us. Understanding and applying new pedagogies, learning new software, identifying the benefits and limitations of new data streams can be overwhelming. And the effects on academia of computing and the Internet are just beginning to be felt. The Decision to Develop Applications in JavaAs we discussed in our last newsletter article (“Jumping into Java,” Unidata Newsletter Winter/ Spring 1999), in response to these challenges and opportunities, Unidata has chosen to attempt developing a new suite of applications, designed by and for Unidata users. The MetApps Task Force, whose members are drawn from among Unidata’s users, was formed by the Unidata Users Committee to work with Unidata’s development staff in creating prototype tools. The decision to develop new display and analysis applications using the Java language represents a fundamental shift in Unidata’s mode of operation. And it carries considerable risk. The long-range goal of this shift is for Unidata users to have a suite of applications designed specifically for their uses, that are capable of evolving in the directions considered important to them, and that are flexible enough to be useful as pedagogies and as research directions change. The risk is that the task may prove too complex for Unidata to achieve with the available resources. “But We Love GEMPAK and McIDAS!”Never fear: Unidata’s support for GEMPAK and McIDAS will continue. The new applications will take time to develop. Indeed, Unidata envisions GEMPAK and McIDAS continuing in their central roles past the year 2003 (the end of Unidata’s current proposal period). These packages, after all, represent the collective repository of the capabilities that we need and use every day. What is unknown, however, is how much these packages will evolve during this period. Enhancements to both GEMPAK and McIDAS are controlled by others (NOAA and SSEC, respectively). Unidata has been informed that, while GEMPAK will continue to evolve for a short time, NOAA intends to cease development of this package when its new software, AWIPS, incorporates all of GEMPAK’s capabilities. Similarly, SSEC has reduced the amount of resources it allocates to McIDAS development in favor of other projects (although support for the package continues at the same level). The realities of waning development of Unidata’s core display and analysis software underscores Unidata’s need to develop its own software. Both Unidata’s staff and the Policy Committee believe that developing new applications in Java represents Unidata’s best hope for the future. MetApps at WorkTo date, MetApps members and Unidata developers have designed, built, and are now testing four prototypes (a surface observation viewer, a model data viewer, a satellite data viewer, and an interactive skew-T). In developing these prototypes Unidata staff and MetApps members are learning not only about building tools in Java, but also how to interact with each other. Unidata has created a Web-based discussion area where “use cases” are proposed and discussed. Use cases are scenarios for how a user would like to interact with data, including what he or she would like the outcome to be. UnidataÕs developers then use these scenarios in developing the prototypes. Inevitably, the process of building a prototype uncovers questions not addressed in the original use case. Users are given the prototype to test extensively. In this process, additional needs become evident. In the next iteration, then, a new prototype is developed. This iterative process should yield a tool that matches with the needs of the users. This is the process that MetApps members are now actively pursuing, and it is educating both the developers and the users. We are very interested in any ideas you may have about capabilities for future applications. You may contact either of us with ideas for assisting in the development of the next generation of analysis and display software: Charles: cmurphy@turbo.kean.edu Russ: rrew@unidata.ucar.edu COMET Case Study Newsby Jeff Weber, Unidata Program Center Hurricane Floyd, which occurred this year on 14Ð18 September, is COMET’s newest case study, CCS020. Floyd, which brought rivers in North Carolina to record flood levels, is the costliest natural disaster for the U.S. so far this year. The previous case, CCS019, deals with the F5 tornado that struck Oklahoma City on 3 May this year. This tornado was well forecast and documented, and it provides a valuable learning tool for major urban tornadoes. The recent release of the Floyd study addresses the National Weather Service’s directive to better quantify precipitation in relation to the underlying topography. In addition, these two case studies represent an effort on our part to distribute significant weather events in the case study format in a timely manner. This allows researchers to access the data sets soon after the event while the research is still current and ongoing. The Case Study Library will continue to offer classic weather cases as well as regional severe weather outbreaks and winter weather scenarios. Data formats continue to an issue. Data in case studies are currently available in the following formats: GEMPAK, McIDAS, NIDS, and GRIB. The two most recent case studies are available as well in netCDF format. The addition of netCDF is necessary to make the case studies compatible with the National Weather ServiceÕs AWIPS platform, and will continue in most future releases. The COMET Case Study Library continues to be augmented by additional data sets such as stream gauge data, radar mosaics, and other data sets and perspectives used in analyzing existing COMET case studies. (See our Web page with links to other case studies: www.unidata.ucar.edu/projects/casestudies/ofInterest/) We strongly encourage you to further enhance the library by contributing your own case studies and/or ancillary data sets. Elizabeth Page of COMET will be presenting a paper, co-authored by Dolores Kiessling (COMET) and me, on the case study effort at the annual AMS meeting in Long Beach in January 2000. The paper will be a part of the AMS IIPS-Unidata Special Session. ACARS Databy Linda Miller, Unidata Program Center Aircraft Communications Addressing and Reporting System (ACARS) data are now available using Local Data Manager (LDM) technology from Forecast Systems Laboratory (FSL) to the requesting site. These data consist of automated weather reports from more than 500 commercial aircraft comprising 50,000 observations per day. The data are useful in aviation and weather research, modeling endeavors, and classroom training activities. A suggested format for referencing the data in presentations or publications might read: Data provided courtesy of six U.S. commercial carriers and NOAAÕs Forecast Systems Laboratory. For detailed information on the data sets, how to obtain the data, and the restrictions on their use, please refer to the FSL ACARS Web page: acweb.fsl.noaa.gov/FAQ.html For information on decoders for use with ACARS on Unidata-supplied application packages, please contact: support@unidata.ucar.edu References: www.unidata.ucar.edu/data/exp.data.html AWIPS and Unidataby Dave Fulker, Director, Unidata Program Center The Advanced Weather Interactive Processing Systems (AWIPS platforms) are deployed at National Weather Service (NWS) forecast offices throughout the country, and many Unidata participants are learning about their extensive capabilities. AWIPS is a powerful tool for integrating meteorological and hydrological data with satellite and radar data, thus providing forecasters with means for issuing more accurate and timely forecasts. Unidata and the Users Committee have followed with interest the development of AWIPS—especially at the Forecast Systems Laboratory (FSL)—and its deployment in NWS offices and at COMET. The Users Committee visited FSL, during its April 1999 meeting, and viewed demonstrations of AWIPS-related systems, FX-Linux, and FX-Net. The FX-Net software extends the capabilities of an AWIPS host server to remote computers running Java. The FX-Linux software is under development and eventually will offer most AWIPS features on PCs running the Linux operating system. By Unidata standards, the costs for AWIPS (as configured for NWS forecast offices) are quite high. Specifically, the Weather Forecast Offices (WFOs) have a satellite downlink and powerful data servers, feeding data over a high-performance local area network to between five to eight workstations. Hewlett Packard computers with 24-bit accelerated graphics cards are required for AWIPS. These costs could be lowered for Unidata universities by using the Linux version, eliminating the graphics cards, and making other compromises. However, substantial design issues also would have to be addressed because most departments would wish to depart from the AWIPS localizations that are used in WFOs. Also, satellite images all are remapped to Lambert Conformal, Polar Stereographic, or Mercator projections. Retrospective data access poses another set of problems. None of these issues is insurmountable, so Unidata faces the question of whether it is wisest to invest in adapting AWIPS to university needs or to proceed along the Java-development path articulated in the most recent five-year NSF proposal under which Unidata is now funded. With the encouragement and support of the Unidata Policy and Users Committees, we have decided on the latter course, though close contact with AWIPS development (at FSL and NWS) will continue. In particular, we intend to achieve data-set compatibility between Unidata and AWIPS, a goal that is facilitated by the fact that AWIPS employs the Unidata netCDF software for data access, and Unidata employs the AWIPS data stream as delivered via NOAAport. (See the accompanying MetApps article on Java development at Unidata, with explicit community involvement.) NEXRAD Data Access and Distribution—
|
|
Please send comments to info@unidata.ucar.edu |
|
   |
Six months ago, we at Unidata lost one of our most brilliant and prolific software engineers. The life of Glenn Paul Davis began near Asheville, North Carolina, on 25 August 1955, and ended on a hillside in Utah, five miles short of the Cedar City airport, on May 3rd of this year. Flying the Mooney M-20 he owned, Glenn radioed his intention to land--though the airport was in the midst of severe thunderstorms, with rain, hail, and low clouds--because he was experiencing icing and a shortage of fuel. Radar and radio contact were lost at 2:53 p.m., and a search was initiated immediately. At 4:00 p.m. a military helicopter discovered the crash site five miles northwest of the airport. Two passengers (Ryan Sanders, of NOAA’s Aeronomy Lab, and Robert Jones, a fellow student of Yoga) lost their lives along with Glenn.
Anne Wilson