Community Newsletter
Summer/Fall 2000

Table of Contents


Shaping the Future: Unidata Users as Leaders, Summer Users Workshop

by Dave Fulker and Jo Hansen, Unidata Program Center

Workshop participants, speakers, and organizers.

Take a week in June; add over sixty workshop attendees meeting in near-perfect Boulder weather. Invite a roster of outstanding speakers and lab leaders to excite, inspire, and motivate thinking and learning. Arrange some light-hearted events where participants can enjoy exchanging ideas in informal settings, and there you have it: a blueprint for a successful workshop.

Unidata (sponsored by the National Science Foundation), and the UCAR COMET program (sponsored by NOAA) jointly funded the workshop, Shaping the Future: Unidata Users as Leaders. The Unidata Users Committee assumed the bulk of the responsibility for defining and organizing the workshop, and the
Unidata Program Office staff arranged the details.

Workshop organizers selected the title: Shaping the Future: Unidata Users as Leaders and settled on four themes:

  • Need and Opportunities for Educational Leadership,
  • Exemplary Uses of Unidata Software,
  • Adventures in Earth Systems Beyond Weather, and
  • New Technologies and their Educational Uses.

The Need and Opportunities for Educational Leadership was the theme for the first day’s plenaries.

Cliff Jacobs, UCAR/NCAR’s National Science Foundation sponsor, led off. Cliff listed the Foundation’s three most visible goals: ideas, people, and tools. Cliff suggested that perhaps the time is ripe for Unidata to contribute to the broader scientific community by actively participating as an essential component in the dialogue to achieve NSF’s goals. At this point the Unidata community possesses the leadership, experience, and tools to play this role.

In the workshop keynote address, NCAR’s new director, Tim Killeen, challenged participants:
To become better equipped to contribute to the important debates concerning resource management, environment, environmental impact, and societal adaptation strategies.

Other energetic, informative, and thought-provoking plenaries followed. On-line teaching modules, interactive homework assignments, developing understanding of how learning takes place, applications for teaching mesoscale modeling concepts, digital library concepts to aid in teaching were among the topics covered. Lab sessions were lively and highly interactive with groups circulating through each of seven labs to learn how to use Unidata tools for teaching and research. Some sessions led participants through exercises playing the student role to encourage greater empathy for them.

Doug Yarger, Iowa State, leads a lab session.

On the fifth and final day of the workshop Unidata Program Director Dave Fulker invited participants to think strategically about Unidata’s future using the experiences of the previous four days. In particular, Dave reiterated points from Killeen’s keynote talk concerning NSF’s hopes for revolutionizing science education “by deploying new interdisciplinary courses and developing curriculum materials that are accessible to and fascinating for science and non-science majors.” To stimulate strategic thinking in the four breakout groups that were about to meet, Fulker asked that the following three questions be considered:

  • Does the Unidata community (perhaps expanded) embrace NSF’s scientific agenda?
  • Is the Unidata community ready and willing to play a leadership role in meeting such an ambitious challenge?
  • If yes (to either), what role should Unidata at UCAR play, perhaps in collaboration with others, such as the DLESE Program Center?

A little social time: Ian Okahe, Charlie Murphy, Arlene Laing, and Oswaldo Garcia.

After hearing reports from the breakout sessions, Fulker synthesized the conclusions as follows:

Unidata users strongly endorse NSF’s new initiatives, as expressed in GEO2000, to improved education in science, mathematics, and engineering and to move to interdisciplinary approaches, and they urge the NSF to configure its grants mechanisms to facilitate university proposals in these areas.

Participants left Boulder with new tools for teaching and research, an enhanced awareness of the challenges facing the Unidata Program Office, and a renewed or enlarged network of professional colleagues.


The Real Time Acquisition and Archival of WSR-88D Base Data

Editor's note: excerpted from an article in preparation for the Bulletin of the American Meteorological Society

by Kelvin K. Droegemeier, School of Meteorology and Center for Analysis and Prediction of Storms, University of Oklahoma

The US National Weather Service recently completed the installation of 120 WSR-88D (NEXRAD) Doppler weather radars in a national network as part of its $4B Modernization and Restructuring Development Plan. (Another 26 Department of Defense and 12 Federal Aviation Administration NEXRAD Radars also were installed.) This unique observing system provides nearly continuous single-Doppler radar coverage across the continental United States which, when coupled with the radar’s superb sensitivity, sophisticated processing algorithms, and advanced user training, has led to a substantial improvement in the identification and short-term warning of hazardous weather.

Although the NEXRAD radars originally were viewed as real-time surveillance systems, their value for scientific research was soon recognized. An interim solution for the long-term archival of full-volume, full-precision base (also known as Level II) data was achieved by outfitting each radar with an 8 mm robotic tape cartridge recording system. Since 1992, the National Climatic Data Center (NCDC) has been making base data available to the national community. Unfortunately, this process is extremely human-resource intensive (6 steps are required to process a single tape), costly (the retrieval of a single data set covering several weeks can cost thousands of dollars), inefficient (obtaining a single data set can take several weeks), and unreliable (the national data archival rate for NWS radars is only 65%, due in large part to the use of tape recording systems that were not designed for continuous use in the field). 

Figure 1. Overlay of NEXRAD radars and Abilene universities. The average distance between a NEXRAD and the nearest Abilene university is only 56 miles.

To provide real time base data for evaluation in storm-scale numerical weather prediction, and to begin addressing the long-term base data archival problem at the NCDC, the Center for Analysis and Prediction of Storms (CAPS) at the University of Oklahoma joined forces in 1998 with UCAR, the University of Washington, the National Severe Storms Laboratory (NSSL), and the WSR-88D Operational Support Facility to establish the Collaborative Radar Acquisition Field Test (CRAFT). Funded initially by a grant from the Oklahoma State Regents for Higher Education, CRAFT is an experiment in the real time compression and internet-based transmission of NEXRAD base data from multiple radars. The initial test bed of radars at Oklahoma City, Tulsa, Fort Smith, Fort Worth, Lubbock, and Amarillo has been delivering data continuously for over a year with virtually no outages. 

CRAFT leverages two important infrastructures to achieve low-cost, reliable transmission of base data in real time. The first is the Unidata Local Data Manager (LDM) software, created by UCAR, which runs on standard PCs or workstations. LDM is used by numerous universities to acquire meteorological data, and also by several elements of the National Weather Service. It has the ability not only to transmit data onto the internet from a given node, but to pass data through from other nodes as well. A notably unique aspect of CRAFT is the addition to LDM of an off-the-shelf, loss-less data compression algorithm, BZIP2, that compresses the base radar data in real time for transmission over low-bandwidth phone lines.

Specifically, base radar data are sent from the radar product generator (RPG) over a high speed serial line to a Sun SparcStation 5 running the radar interface data distribution system (RIDDS) software developed by the NSSL. The data are output in UDP packets to an 8 port Ethernet hub. The CRAFT system involves linking to the RIDDS Sun workstation an inexpensive ($1,500) personal computer running the Linux operating system and LDM. This PC requires only nominal memory and disk space (64 megabytes and 2 gigabytes, respectively), and is purchased without a monitor. The data then travel from the PC to an optional $2000 multi-port router and then onto a 56 kilobit/sec phone line.

BZIP2 compresses the base data in real time down to an average of 1/12th their original size, and in light of the fact that a single 5- or 6-min volume scan is typically a few megabytes in size and never exceeds about 15 megabytes, the data compression achieved is more than adequate for a 56 kilobit/sec phone line, even in the most extreme situations (e.g., a hurricane). Indeed, the aggregate compressed base data rate for the entire national WSR-88D network is only 30-40 megabits/sec, and consequently bandwidth is not an issue. Data decompression is performed in real time at the recipient end, and in the event of communications failure, the LDM personal computer at the radar site will store and then re-transmit up to 4 days worth of data. This amount can be increased by increasing the capacity of the local disk.

In early June, the NCDC began receiving compressed base data in real time from the 6 CRAFT radars via the commodity internet. By early August, these data were being transferred directly and automatically, to the NCDC mass storage system. According to Steve DelGreco, the compressed base data stream on the community T1 internet line represented only a “trickle” compared to available bandwidth. Efforts are now underway to develop and test an automated retrieval system whereby users can query the NCDC data base and download compressed, archived base data via the internet. Additionally, visualization software is planned to be made available so that users can browse files on the web prior to download. 

Recently, CAPS, NSSL, and the NCDC were awarded a NOAA Environmental Services Data and Information Management (ESDIM) grant to expand the successful CRAFT concept for eventual application to the entire NEXRAD network. This new effort, known as CRAFT-2, takes advantage of two major national networking infrastructures: Internet2 and Abilene. Internet2 is a consortium of nearly 200 universities involved in developing new tools and applications for the Next Generation Internet, and Abilene is a high-capacity network backbone that supports these efforts. 

Figure 2. Logical networking topology for CRAFT-2.

An overlay of Internet2 universities and the NEXRAD network reveals that the average distance from any radar to the nearest Internet2 node is 56 miles. (Figure 1) By running a 56 kilobit/sec phone line from each radar to the nearest Internet2 node via collaborative arrangements with regional networks, the CRAFT concept is immediately reproducible, at relatively low cost, nationwide. Once the base data arrive at an Internet2 site, they can be transferred to the high-speed Abilene backbone where they can be made available to all users. To access data from a particular radar, a user will simply enter the appropriate radar LDM/IP address into their local LDM system. (Figure 2)

Selected sites on Abilene, such as high-priority NOAA facilities and universities, will serve as transfer points for the entire data stream, while “satellite” nodes not linked directly to the Abilene network will obtain base data via the commodity internet for as many radars as available bandwidth allows. As networking capabilities in the US continue to expand, bandwidth limitations are likely to vanish entirely. 

During the next 12 months, CAPS and the NSSL plan to add approximately 30 RIDDS-equipped radars to the Internet2/Abilene infrastructure as part of CRAFT-2. The broad geographic distribution will provide an acid test of overall reliability, network efficiency, and real time ingest at the NCDC (and, eventually, NCEP). As part of this effort, NSSL and the University of Oklahoma will improve the radar data compression algorithms with a view toward accommodating larger data sets associated with dual-polarization and more dense scanning strategies. Attention also will be directed toward implementing the LDM-based data compression and transmission capabilities in the new NEXRAD Open Systems architecture. 

Real time base data will be of rather limited value if not accompanied by suitable analysis tools. Consequently, preliminary efforts are underway to explore the application of data mining techniques to base data, and the creation of synthetic climatologies and other metadata sets by running storm feature identification algorithms on the data as they arrive from the radar. Consideration also is being given to creating one or two sites that would maintain 2 or 3 years worth of base data online for immediate perusal and download.


A New Unidata LDM: Scaling Up for a Data Deluge

by Russ Rew, Unidata Program Center

Unidata is testing a new version of the LDM (Local Data Manager) that has the ability to handle data at a significantly higher rate than the current version of the software. Below we describe scalability problems with the current version and a little of the computer science that went into the design of the new version.

Although the current LDM has been dealing with the load of distributing data on the IDD (Internet Data Distribution) system, scalability problems loom on the horizon. Before explaining those problems, we first need a little terminology. 

A data product is the smallest unit of data dealt with by the LDM, and represents a single named data item, such as the report of an observation, a gridded data field, or an image. Data products can be as small as a few dozen bytes for a short bulletin or as large as many megabytes for a satellite image. The data streams commonly distributed to most IDD sites
comprise almost 500,000 products per day, making it impractical to store each product as a single file. Data products from new data sources (radar data, model output, imagery) will increase the daily stream to nearly 1.5 million products by the end of this year. 

Each LDM system has a local data store (implemented as a single large file) called the product queue, which is shared among the LDM programs that get data from upstream hosts, read and process the data, and send data on to downstream hosts. Old products are expired out of the product queue when they are no longer needed. Product queues typically hold
about an hour’s worth of data from most data streams, though they can be configured to hold data from different data streams for different intervals. 

The amount of time it currently takes to insert or delete a product is dependent on the number of products in the product queue. When product queues are smaller than about 10,000 products this time is negligible, because the coefficient of the linear term is small. However, as the rate of data products has increased, this term has become very significant, to the point
that a current LDM system may have difficulty keeping up when it has 50,000 or more products in its queue. This results in delayed data to downstream sites and even data loss if there are sustained periods during which the LDM cannot insert data into the product queue as fast as it becomes available. At the Unidata Program Center we saw a dramatic demonstration
of this problem when we tried running an LDM system to inject all data products, including radar data and some experimental data streams not yet generally available. Pqexpire, an LDM program that was supposed to delete old products from the queue, could not delete them as fast as new products were added, so data products were lost for lack of space in the queue. 

The problem of designing a data structure to support the operations of search, insert, and delete efficiently is a classic computer science problem. Solutions include “balanced-trees” such as the AVL tree, the B-tree, and the red-black tree. In designing the new LDM product queue, we chose a relatively recent computer science discovery, the skip list that provides a simpler and faster way to achieve high performance by relying on “probabilistic balancing.”

The results of adding this technology to the LDM are dramatic improvements to the time needed to insert, delete, and find products in a large product queue. For example, it takes only 15 seconds for the new LDM compared with over 12 minutes for the current LDM to delete the products out of a test queue. 

There are two additional benefits of the new LDM design. First, it is no longer necessary to run the pqexpire program, which is the worst current bottleneck when there are a lot of products in the queue. Although pqexpire still works with the new version, we recommend that new LDM systems be configured to run without pqexpire, deleting oldest products as
needed when new products arrive. Second, there is no longer a need to specify an arbitrary limit, such as one hour, for the time that products will reside in the queue. The size of the queue specified when it is created will determine how long products will stay in the queue, because they will only be deleted when the space they occupy is needed for new products.
This will provide additional elasticity for the IDD and will mean that a significant amount of data remains available for processing even if connectivity is lost to upstream data sources.

An additional LDM improvement is much larger product queues. Currently, product queue sizes are limited to about 2 Gbytes because of the signed 32-bit type used to represent a file offset on most systems; however, we have eliminated this limit to support much larger product queues by modifying the LDM for use on architectures that support 64-bit file offsets.
With larger product queues, a data archive could conceivably be represented and accessed as an LDM product queue providing a convenient form of retrospective data access for other LDMs. 

When the new version of the LDM is available for testing, an announcement will appear on the ldm-users@unidata.ucar.edu mailing list. The improvements in scalability should arrive in time to deal with the large increases in the number of data products available early next year, when NIDS radar products are added to the NOAAPORT data stream.•

References

W. Pugh. ``Skip Lists: A Probabilistic Alternative to Balanced Trees’’. Communications of the ACM, vol. 33, p. 668-676, June 1990. 


MetApps Project Update

by Charlie Murphy, Kean University and Don Murray, Unidata Program Center

What’s New?

In the nearly two years since its inception, the MetApps Project has made significant progress. The Development Team has built the following four prototypes:

  • Surface Observation Viewer
  • Interactive 3D Sounding (Skew-T) 
  • Model Data Viewer
  • Satellite Image Viewer

Development on the Surface Observation Viewer has ceased. Lessons learned in developing that prototype and some of the components have been incorporated into the remaining prototypes. The other three prototypes are in active development and testing at this time. Anticipating the cessation of development of some of the current prototypes, the Task Force is currently developing use cases (scenarios for using the software) for a tool to display and analyze radar data, including Level II data. 

In addition to writing use cases and developing and testing prototypes, the participants in the MetApps project are also learning how to interact and communicate with each other. The evolution of communication and discussion mechanisms has been a significant MetApps accomplishment over the past two years. UMADA (see below), as the central information and discussion tool has matured and gained acceptance among MetApps personnel. Communication in an electronic mode, although efficient, often misses important contributions and ideas. Recently the Task Force has explored telephone conference calls as an additional way for the Task Force to communicate. Reaction to using conference calls has been very positive and will likely be included as a regular part of the Task Force operations in the future. One of the most important lessons over the past two years is that the success of a development project is very dependent upon its communication structure.

Three new members joined the Task Force this spring: Chris Herbster (Embry-Riddle), Bob Ross (Millersville), and James Kelly (Australian Bureau of Meteorology). At the UPC, a new staff member, Stu Wier, was hired for MetApps development. Murray was named the new MetApps Project Leader, replacing Russ Rew, who will continue work on LDM scalability enhancements.

What’s UMADA? 

Central to the MetApps process is its communication structure. Facilitating communication and maintaining a record of transactions is essential to the success of the project. Early on, e-mail was used for communication between Task Force members and the Development Team. It quickly became apparent that e-mail has a number of drawbacks for this particular use. The introduction of a Web-based discussion tool, known as UMADA (for Unidata MetApps Discussion Area) provides a much more satisfactory mechanism for communication and feedback. 

UMADA is based on the Digital Document Discourse Environment (D3E, a collaborative project started at the Knowledge Media Institute at the Open University in the United Kingdom). The D3E approach assumes that a document can serve as a starting point for ?he discussion process, and the process can move the group toward building a consensus. With UMADA the discussions are tied to particular documents producing a directed debate focussed on the specific needs of the MetApps project. As prototype tools are refined the UMADA documents and discussions provide a “group memory” of the design and implementation process that can be referenced in later work. You can learn more about UMADA and review the current discussions from the MetApps homepage.

MetApps in the Public Eye

At the Unidata session of the AMS IIPS conference in Long Beach, Russ Rew presented a talk on the MetApps project and the component design that is being used in developing the prototypes. Don Murray gave a demonstration of several features of the prototypes during this talk. Don also presented a talk on the MetApps project at the 20th Northeastern Storm Conference where the audience oo-ed and ah-ed at the 3-D hodograph and the rotating globe of animating satellite images. A recent article by Sun on their Java web site featured the MetApps project in a discussion of Java 3D and scientific visualization.

At the recent Unidata Summer workshop, a half-day was devoted to the future of Unidata applications and covered topics such as Java, VisAD and the MetApps project. Charlie Murphy gave a talk “MetApps: Building Tools for the Future” which discussed the whats, whys, and hows of the project. The audience was informed of the project’s background and how the task force and the developers work together to design, develop and test the prototypes. A two and a half-hour lab session allowed participants a chance to become familiar with the prototype applications. They were given an overview of the features of each prototype and an overview of the UMADA discussion area. Part of the time was spent using the Interactive Sounding Application as a tool to demonstrate how this type of application could be used to replace paper-based exercises to identify common features of atmospheric soundings. The rest of the time, participants got some “hands-on” time to use and evaluate each of the applications. The prototypes received favorable reviews and many suggestions for new features were gathered.

What’s Next?

The next round of use cases will focus on the display and analysis of radar data for research and teaching applications. These use cases will be out in mid-July and the developers are already working on the underpinnings for handling these data.

New versions of the Interactive Sounding Application and Gridded Data Viewer were recently released and a new version of the Image Viewer will be out by late summer.

How can I get involved?

Communication with the User Community is vital for the success of this project as well. One of the reasons for the existence of the MetApps Task Force is to provide a small representative group that can efficiently communicate with the Development Team. It was considered to be difficult and premature to open up discussion to the entire Users Community at the early stages of the project. At this point broader user input can be gained through workshop sessions where the prototype tools are demonstrated and users can work with them directly. Also, at the current time, UMADA provides free and open access to all of the discussion and for access to the prototypes as well. Anyone can download, install and try out the existing prototype applications from the MetApps web page. If you want to provide feedback on a particular application or you want to compliment the Development Team on the great job they are doing, you can contact any of the Task Force members and pass along your thoughts and comments. In the near future UMADA will allow any user to post comments in a user’s discussion area, which the Task Force will monitor for incorporation into their work.


Spring Policy Committee Meeting: Whither Unidata?

by Unidata Staff 

Unidata Policy Committee meetings typically are low-key affairs. The Committee convenes, transacts business, debates issues, and makes recommendations. The resulting changes in policy, even rather significant ones, usually occur in small steps that allow the program and its community to roll along, making comfortably incremental changes when need dictates.

Intermittently, however, the group’s conclusions are far-reaching and represent major changes in direction, as occurred when the Committee recommended that Internet Data Distribution (IDD) be developed to replace the satellite broadcast mode of data delivery throughout Unidata. A recent meeting in Madison, Wisconsin, (21­22 May) also had the air of such change. Strategic planning was the principal agenda item, motivated by several factors:

  • Opportunities and requests for collaboration are increasingly numerous, and the Policy Committee concluded, at its winter meeting, that Unidata should evaluate potential collaborations in the context of an overarching strategic plan.
  • By many indications, the interests of the Unidata community are expanding in respect to educational methodologies and interdisciplinary science, and these changes have implications for the data tools that Unidata offers.
  • UCAR’s new Program for the Advancement of Geoscience Education has gained significant funding to help build the Digital Library for Earth Science Education, has renamed itself the DLESE Program Center, and is interacting very closely with the Unidata Program Center in several aspects of the endeavor.
  • New leadership and recent strategic planning efforts (the GEO 2000 document, e.g.) at the National Science Foundation (NSF) are changing the context in which Unidata and its community members may gain federal funding.

Committee members, UPC staff members, agency representatives, and invited observers participated in intense debate and discussion for an entire day, setting the stage for adopting a 5­10 year strategic plan at the fall meeting. Helping to launch the discussion, Cliff Jacobs presented his views on how key initiatives at the NSF, especially at the level of the Geosciences Directorate and the overall NSF Director, offer Unidata opportunities to contribute to these central science themes. His comments highlighted the uniqueness of Unidata, including its strengths in utilizing technology to advance education and research and its success in engaging community members as genuine contributors. Cliff challenged the group to recognize its responsibility for helping other communities achieve similar benefits from technology.

A draft (strawman) strategic plan served as the basis for the day’s deliberations. Developed over months of spirited discussion among Unidata staff, the strawman emphasized an interdisciplinary (i.e., Earth-system oriented) posture for Unidata and set forth the following long-range goals:

  • Real-time, self-managing data flows,
  • Software to visualize/analyze geoscience data,
  • Distributed, organized collections of digital material,
  • Computer-augmented discourse and discovery, and 
  • Support and engagement of diverse constituents.

The draft assumed that Unidata’s constituency would remain academic and that its mission would continue to include an emphasis on building community.

Program Director Dave Fulker asked attendees to focus on long-term strategies that will keep the program flexible and prevent being “overtaken by events.” Before dividing into smaller groups, participants were challenged by Fulker to consider a revised Unidata mission: “Providing infrastructure and software for use in academia to observe, understand, and share knowledge about Earth systems.” He also urged thinking about the new NSF/GEO science agenda, which highlights four areas of study—planetary structure, planetary energetics, planetary ecology, and planetary metabolism—without even mentioning the traditional disciplines that comprise the geosciences.

Breakout-group discussions were vigorous, and each group’s summary of its conclusions contained both significant similarities to and significant differences from the other groups’ summaries. To a degree that was surprising to some, there was general agreement on the strawman goals and on the merits of a more interdisciplinary Unidata, with a stronger role in supporting educational change. However, none of the four breakout groups was satisfied with Fulker’s new mission statement, so four potential replacements were drafted on the spot!

Other issues raised include the following:

  • How quickly can an expansion of Unidata’s role occur without detrimental effects on the community presently served?
  • If there is increased emphasis on serving community colleges, could the support responsibilities be handled by a subcontract or other mechanism to reduce the impact on staffing at the Unidata Program Center? 
  • If the expansion has an international component, should countries with restrictive data polices be treated differently than those with open policies?
  • Will greater distinctions arise between educator and researcher data needs, with Earth-system educators needing to integrate a larger variety of data types?
  • If the expanded role embraces more “data on demand” (i.e., archived data, with which the program has relatively little experience) can Unidata work jointly with NCAR’s Scientific Computing Division to articulate a vision that fulfills the retrospective data needs of the (broader) community?
  • While striving to insulate a more diverse community of users from the complexities of technologies and data sets, can Unidata successfully leave the content-explanation role to the universities, where it belongs? 

While reaffirming Unidata’s commitment to high-level support for its present community, the Policy Committee agreed to further study an enlarged scope, designed to serve increasingly complex and interdisciplinary scientific and educational needs in the university community. The fall meeting will focus on implementation issues and formally adopting the strategic plan. Readers with thoughts on this matter are encouraged to contact members of the Policy Committee.


Getting Started with DODS

by Ethan Davis, Unidata Program Center

The Distributed Ocean Data System (DODS) project has grown. Just in the past year, the number and variety of data sets available through DODS multiplied considerably. To track them, we revise and maintain a DODS Dataset web page on an ongoing basis.

(Please let us know if you are serving any data with DODS and the data set is not listed. We want the list to be as comprehensive as possible.)

Access to DODS Data

Among the several ways to access DODS data, the simplest is to use a web browser. First, go to the above-referenced DODS Dataset Page, and follow the links provided. The links will give you information on the datasets in two different formats.
A second method for accessing the data is to use the DODS web interface available from newer DODS servers. The DODS web interface is a good tool for developing a better understanding of DODS URLs. For instance, http://dods.gso.uri.edu/cgi-bin/nph-nc/data/ lists data files and directories. If you click on a dataset, you will get a web form that lists the attributes and variables that make up the dataset. The URL for that dataset is shown at the top of the form. As you click on desired variables and change the range of the data to be accessed, the DODS URL at the top of the form reflects those changes. The result is a constrained DODS URL. Select the “ASCII” button on the form to view the data in a comma-separated ASCII format. Or, cut and paste this URL into a DODS client application. With a slight modification, you can use the resulting URL to pull the data into a spreadsheet like Excel. Simply add the string ‘.ascii’ immediately before the ‘?’ in the URL, and in Excel “open” that URL (be sure to use the correct file type-text, comma-separated values). 

The long-term goal of the DODS project is to allow users to access remote data using the data analysis applications with which they are comfortable. For example, a Ferret user, can start accessing DODS data immediately by simply giving Ferret a DODS URL instead of a local filename. A Matlab user can download DODS software that allows access to DODS data in Matlab. If your application uses the netCDF API to access data, you can make your application a DODS client by relinking it with the DODS netCDF client library. 

These are the primary applications that DODS currently supports as clients; however, a number of groups are working on other DODS clients. So if you are interested in using DODS with other applications, contact us (see below). We’ll let you know if anyone is working on that application and, perhaps, how you can join in the effort.

Coming Soon

A number of components will be available soon:

  • An IDL DODS client,
  • The DODS library in Java,
  • An SQL server, and
  • A GRIB server.

Data from the Unidata IDD also will be available. NCAR/SCD and Unidata are working together to make archived IDD data available to the Unidata community. DODS servers will be one of the tools available to allow users to access this data.

Contact Us

If you have any questions, please email us at support@unidata.ucar.edu.


COMET Case Study Update

by Jeff Weber, Unidata Program Center

The COMET Case Study library continues to expand and now offers 26 complete meteorological data sets at no charge. These data sets consist of NCEP model ouput, NIDS and NOWRAD radar data, surface and upper air observations, weather advisorie and bulletins, and satellite imagery in various wavelengths and resolutions. The library has achieved large growth in not only the number of cases offered, but also in the amount of data downloaded, and number of users. For a detailed analysis of these parameters, please see the latest executive summary:

http://www.unidata.ucar.edu/projects/casestudies/execSummary/200004/

The library continues to enhance the datasets with new data and has begun to add ACARS data to cases that occurred after May 1999. The ACARS data sets will be available in the native netCDF format as well as being decoded on-the-fly into GEMPAK files to allow greater use of these data. The latest five cases to be made available are:

  • Pacific Northwest Winter Storm, 23­26 November 1999, COMET Case Study 026, 
  • Salt Lake City Tornado/Long Island Flash Flood, 11­12 August 1999,COMET Case Study 025, 
  • Explosive East Coast Cyclogenesis, 19­26 January 2000, COMET Case Study 024, 
  • Winter Severe Weather, 08­11 November 1998, COMET Case Study 023, and 
  • Northeast US Severe Weather, 02­03 June 1998 COMET Case Study 022. 

For access to these cases see: http://www.joss.ucar.edu/cometCases/

Contributed cases or suggestions for future cases to be included in the library are always welcome. We appreciate any contributions of labs or exercises based on existing COMET case studies. We encourage you to join the case studies mail list so you can be informed of any changes or enhancements to the library. 

To subscribe please visit: http://www.unidata.ucar.edu/mailinglist/mailing-list-form.html and subscribe to “casestudies” and any other list you are interested in. To see what the future holds for the library visit our status page (web page no longer available) which lists all case studies scheduled for publication.


A Delay in Free Access to NIDS Radar Data via NOAAport

by Ben Domenico and Linda Miller, Unidata Program Center

In July 2000, the National Weather Service (NWS) announced that the free distribution of the NIDS (NEXRAD Information and Dissemination System) data stream, via NOAAport, would be delayed through 31 December 2000. So Unidata universities waiting to receive free NIDS products via the IDD (Internet Data Distribution) system will have to wait a little longer for the data. Of course, the data are still available from the three NIDS vendors for a fee, so the current NIDS subscribers will need to renew their contracts, at least through December.

NIDS data are collected from 153 WSR-88D radars in the U.S. and Puerto Rico. The radars are part of the extensive NWS modernization and are funded by the NWS, the Department of Transportation (DOT), and the Department of Defense (DOD).

Unidata is prepared to make the data available to its participants, via NOAAport using the IDD/LDM technology. What does this mean to the Unidata university community? A broader use of radar data by universities and colleges which previously were unable to subscribe and pay the fees associated with the NIDS. The NIDS data (which are Level III products) will be distributed through NOAA-port using the IDD. These data will form a subset of the full NIDS distribution; they will be the same data set defined by the NWS forecasters as “required data.” Testing will begin in late summer in preparation for the transition. Among other options for access to the data are FTP servers located at the NWS Office of Systems Operations in Silver Spring, Maryland.

In addition, Unidata is coordinating with the University of Oklahoma and other groups to access and distribute the raw volume scan (Level II WSR-88D, as opposed to NIDS products) radar data, which are proving to be valuable for research, modeling and other applications. See Kelvin Droegemeier’s article (page 3 describing Level II data and the CRAFT project.

For information on product availability, including the subset, see: NWS “Dissemination of WSR-88D Products Concept Paper:”(web page no longer available).

Other references:

Radar Product Central Collection/Distribution Service

http://www.nws.noaa.gov/tg/rpccds.html

NOAAport Users Page

CRAFT Home Page


A Fond Farewell to Bob Fox

by Jo Hansen, Unidata Program Center

Bob Fox shows off one of the many tributes he received.

The Unidata Program Center and its Policy Committee honored Bob Fox at its spring meeting. Bob, more formally known as Dr. Robert J. Fox, retires this summer after twenty years as Executive Director at the University of Wisconsin-Madison’s Space Science and Engineering Center. His retirement from the University coincided with the conclusion of his fourteen years on the committee, seven of which were as its chair.

We honored Bob with a lively, entertaining evening featuring a combo, a trumpet solo improvised by Unidata Program Director Dave Fulker, and roasts and toasts by many participants including the presentation of a plaque from the National Climatic Data Center with a certified copy of the weather report on the day he was born in Michigan. Good wishes were sent by others unable to attend, including UCAR president, Rick Anthes, who wrote:

Bob’s intelligence, common sense, and sense of humor always impress me. He has an unusual ability to see the big picture and the long-term vision as well as what is needed in the short term to realize that vision. Thanks for all you have contributed, Bob!

Left to right: Mary Marlino, Bob Fox, Jean Fox, Charlie Murphy, Mohan Ramamurthy, and Russ Rew.

While Bob’s humor, wisdom, discernment of key issues, and skill in chairing the Polcomm were keys to its success, his contributions did not stop there. As a committee member his experience, and insight illuminated many a thorny issue. Although his body lýnguage suggested (occasionally) that his mind was not on the business at hand, he would stun the collective group by suddenly sitting bolt upright and placing an insightful, concise, and often controversial proposal on the table for consideration. Of considerable significance to Unidata and its community, Bob was the major player in helping Unidata utilize the downlinking and data processing services at SSEC to generate data streams of value in university education and research. Or, as one former very “punny” Policy Committee member (Jim Moore) wrote: “He was instrumental in getting satellite data to universities but never feared to speak his mind; no one would ever censor him,” or “People just could not understand his work with satellites as it was over their heads.”

While acknowledging his technical and scientific contributions many also commended Bob for his generosity, compassion, and fundamental decency. This is an uncommon man.

We’ll miss you Bob.


Staffing Changes: Introducing Stuart Wier

by Jo Hansen, Unidata Program Center

Some of you out there in Unidata-userland have met the latest addition to the Unidata staff: Stuart Wier. For those who haven’t, allow me to introduce him.

Stu came to Unidata in January of this year after 13 years at NOAA’s Forecast Systems Lab (FSL) where he developed a system to display meteorological data from numerical weather models for the AWIPS WS. Since these data are similar to the model data on the LDM, the present assignment in the Development and Support Group at Unidata, to develop Java applications for MetApps, is a natural progression.

Stu is enthusiastic about Unidata, his position, his colleagues, the almost-academic feel of the program office, and the independence his assignment allows. 

Stu studied Physics at Duke University and received his PhD in Geophysics from Princeton.

When he’s not at his desk at Unidata, Stu enjoys spending time with his family: daughter Sally, wife Pat, and son Tom who is a member of the U.S. Canoe and Kayak team, but will not travel to Sydney since his kind of canoe race is not in the Olympics.

And, in case you’ve spotted a spiffy, green, vintage MG in the Foothills Building Four parking lot and wondered who the lucky owner might be, wonder no longer. Check the photo.

Please send comments to info@unidata.ucar.edu