Table of Contents

• NetCDF: Achieving Transportability of Scientific Data
• USSDM Aplications Task Force Formed
• Director's Report
• Manager's Report
• Workshop on Synoptic Meteorology Instruction
• McBE Committee Meeting Summary
• USSDM Workshop Scheduled
• New PC-McIDAS Release
• Unidata in the Literature

NetCDF: Achieving Transportability of Scientific Data

The Unidata local data management (LDM) system provides a means for capturing and decoding current weather data from broadcast. "Decoding" implies transforming something cryptic into something useful, namely, into a form that can be used later for an application.

In many systems, the data structures are specific to a given data set, and any analysis software must incorporate specific knowledge of this data structure into its design. This leads to a superfluity of analysis applications that perform similar functions on differing data sets. Years of programmer time are spent writing "front ends" to read various data sets.

Personnel at the National Space Science Data Center (NSSDC) of NASA Goddard Space Flight Center were faced with data from a plethora of scientific experiments from a variety of missions. (For a full explanation, see Lloyd Treinish's article in the 14 July 1987 issue of Eos.) Starting in about 1982, NSSDC developed a "data-independent" self-describing data structure and associated software. Some goals of this design are as follows:

• Provide for storage and random access of multidimensional data whose representation may be of various types (such as integer, real number, text, or raw bytes).
• Provide for description of the data within the structure (i.e., are these numbers temperatures in degrees Celsius, or what?)
• Provide a means for decoupling analysis considerations from data-storage considerations. This means abstracting applications programs from the details of data storage and retrieval.
• Provide an abstraction that is oriented toward the scientific user of the data.

These goals match the Unidata goals. In particular, using a self-describing data structure allows us to design powerful display and analysis systems that will be useful across diverse data sets.

Additional Unidata goals affect the internal data structures but not the interface or data abstraction. In the initial NSSDC environment, all the target machines were VMS VAXes. In the Unidata workstation environment, data is shared among computing machines that have potentially different architectures. Two scientists may have machines with differing representations of floating-point numbers. In modern local area network environments, they may share the same storage media resources via a network file system. Hence, we have added the design goal:

• Provide data storage and access uniformly across machine architectures. Allow data files produced on one machine to be copied via simple means (byte stream transfer) to another and be accessed there.

An approach to this goal is to use the External Data Representation (XDR). (See the Sun Microsystems 0S release 3.4 documentation in 1988 entitled "External Data Representation Protocol Specification.") XDR is a standard for describing and encoding data primitives such as integers, floating-point numbers, and characters. It also provides for counted arrays, structures, and discriminated unions composed from the primitives. (For example, an image might be represented as a counted array of bytes.) It is a "working standard"; it has been shown capable of communicating data between such diverse machines as Sun Workstations, VAXes, IBM-PCs, Crays, and (recently) IBM mainframes. (See Bob Bramwell, "Linking an IBM Mainframe to Several Sun Systems through RPC," Sun Technology, Spring 1988.) Source code for C-language implementations of XDR is in the public domain.

These two proven systems combine to form the Unidata netCDF. The same 13 VAX Fortran function calls in use at NSSDC could be used as the interface in a VAX Fortran environment. (See Michael Gough, CDF Implementer's Guide, National Space Science Data Center publication NSSDC/SAR 87-01, 1987) This essentially dictates the form of the C-language interface and data structures used. The UNIX f77 Fortran interface parallels the VAX Fortran interface as much as possible. The current prototype netCDF meets the above criteria. It exists as a C-language interface supporting an XDR file structure specification. The prototype has been tested on a number of architectures including Sun 3, Pyramid, VAX Ultrix, and VAX VMS. Data files have been shown to be freely exchangeable among those machines.

We caution the reader regarding potential problems or shortcomings of the netCDF.

• The Fortran interface "overloads" some of the types of certain arguments. That is to say, the same argument position of a function call may be used to pass an integer value at one place in a program and a floating-point value in another. This is breaking the rules for the sake of a simpler interface and may be impossible to implement in some environments. (This is not a problem with the C interface.)
• The CDF is not a database system. Data values are referenced with name and index, like "TEMP 1, TEMP 2, ..." as opposed to "the TEMP at TIME 3:15 is ____".
• The netCDF is optimized for data access, not for storage space. (For example, the data are not packed.)
• Use of netCDF requires more overhead than use of data-specific structures or data-specific XDR structures.

In addition, the current prototype makes heavy demands on virtual memory.

The UPC is currently adapting decoders to use netCDF. The Fortran interfaces are also still under development, but near completion. Our plans specify that Unidata analysis applications will use netCDF to access data captured from the Unidata local data management system.

USSDM Aplications Task Force Formed

The Unidata Implementation Working Group (IWG) has agreed on an implementation plan for the Unidata System for Scientific Data Management (USSDM, formerly called the UNIX/VMS system). The plan encompasses the integration of (1) the local data management (LDM) system, which as been running at several test sites for several months; (2) the Unidata network implementation of NASA's Common Data Formed (dubbed netCDF) for storing data in a portable format; (3) a set of decoders to take the data ingested by the LDM and decode and store them in network accessible netCDF files; and (4) a set of applications to allow users to access, analyze, and display the data on a heterogeneous network of workstations.

Having agreed on the long-term plan for the USSDM, the IWG endorsed a proposal to establish a Unidata Program Center (UPC) task force to implement the system. Specifically, the task force will integrate the applications suite (including image processing) with the decoders, netCDF, and LDM. The initial release of the system will take place at the June workshop. The remainder of the system is to be implemented during Phase IV of Unidata's program plan, which schedules roughly three releases per year.

The task force comprises three members of the UPC staff and three visiting scientists. (The visiting scientist positions are rotating appointments made by the UPC director from the community at-large.) The visiting scientists may, in fact, work from their home institutions, employing electronic data networks for day-to-day communication. The UPC staff members on the task force are Bob Green (image processing), Russ Rew (netCDF), and Tom Yoksas (scientific applications and task force leader).

The Applications Task Force will establish a "look and feel" for the USSDM, standardize its development environment, document file data structures, and publish design specifications. Its charge is to design the applications layer of the USSDM with extensibility in mind so that new, emerging technologies in hardware and display software can be accommodated in the future.

The foundation of the applications layer will be designed to handle generic data types so as to accommodate as many scientific data sources as possible. This will ensure the long-term utility of the USSDM. Generic data types identified to date include point observations (typified by surface and ship observations), soundings (typified by upper air observations), gridded fields (typified by numerical analyses and forecasts), and images (typified by satellite observations and radar measurements). Input media will include high-speed communications circuits, access to local data and network databases, and CD-ROMs.

The applications suite will be designed to encompass image processing. Design factors will include image display, animation, navigation for satellite imagery (projection and remapping), graphical overlay capabilities, color manipulations, zooming and panning of images, areal statistics, frame saving and restoration, and image inversion. An Image Processing Specifications Committee will be formed to define community needs. Also included in the USSDM suite of applications will be a basic set of mathematical and data-management operators for the manipulation of netCDF data files. Again, university input as to needed functionality of the operators will be solicited by the task force after the initial release of the USSDM in June.

Because of the potential wide-spread and diverse use of the USSDM, issues of software quality will receive careful attention by the task force. Software routines that will be integrated into the USSDM will be judged on usefulness, scientific correctness, code quality, documentation, modularity, extensibility, availability of resources to integrate the software, and ease of maintenance.

The Applications Task Force will look to universities, government agencies, and possibly commercial and private sources in addition to personnel at the UPC for contributions (of both software and integration resources) to the USSDM software package. Several potential contributors have already been identified and additional sources will be sought to ensure that new ideas are incorporated in the USSDM. As the USSDM evolves, additional processing capabilities will be added as recommended by the user community at-large.

USSDM Applications

After considerable discussion, the IWG divided the application functions into three priority categories: high, medium, and long range. In the high-priority grouping are functions such as contouring of NMC grids, surface observations, upper-air observations, and derived data; thermodynamic diagrams from upper-air observations; and data plotting at stations or gridpoints of NMC grids, surface observations, upper-air observations, and derived data. Of medium priority are such applications as deriving kinematic parameters from surface and upper-air observations and deriving local scalars for and time differencing of NMC grids and surface and upper-air observations.

The matrix has undergone several revision by the IWG, including changes in the priority ranking of the various elements, with significant comment by participants from New Mexico Tech, Pennsylvania State University, and Purdue University. The demonstration set for the initial release was chosen on the basis of the matrix and a practical consideration: was the university offering the application also able to offer some of the manpower necessary to adapt the application to the LDM? Applications contributed by Purdue met both criteria. (See the USSDM Workshop announcement for the complete list of functions in the initial release.)

Director's Report

We anticpate that Phase IV, the deployment phase, of the Unidata program will soon begin officially. (We are expecting the a grant to UCAR for this purpose will by made by the Atmospheric Sciences Division at NSF in the near future.) We actually began Phase IV in February when the Unidata Program Center (UPC) conducted its first PC-McIDAS training workshop; at that time, however, we were still continuing the Phase III activities of developing software for computers running the UNIX and VMS operating systems

The first release of this UNIX/VMS software, which we have named the Unidata System for Scientific Data Management or USSDM, is now essentially complete; we have scheduled a workshop late in June for training users on it. At that point we will be fully engaged in Phase IV activities.

As I think back over the preceding phases of Unidata, I am impressed by the quantity and quality of university contributions to the endeavor. The persistent enthusiasm for Unidata evidenced by these contributions, many of them made without reimbursement, has made the program more of a grass roots effort than I would have dared to guess four or five years ago. I am greatly indebted to all of you who have offered your guidance and your resources.

I could make a contributors list of great length, but I would like to single out three institutions for their contributions toward recent Unidata progress:

• The University of Maryland's George Huffman was a prolific and valuable contributor to Unidata's Local Hardware and Software Systems (LOHSS) Working Group in Phase II. More recently he has played an important role in specifying functionality and other characteristics for the USSDM and the local data manager (LDM) contained therein.
• Purdue University has been a key contributor since Unidata's inception, beginning with Ernest Agee's participation in the Steering Committee and his chairing of the LOHSS Working Group. Software developed at Purdue, by Daniel Vietor, Ross Aiken, and others, has followed Unidata standards and other technical concepts to a considerable degree, and we recently benefitted from these characteristics when we incorporated the Purdue software into the USSDM initial release with relative ease.
• The University of Wisconsin has been a leader in interactive data processing and display since more than a decade before Unidata existed. This leadership continues as Robert Fox, Tom Whittaker, J. T. Young, and others have developed the Unidata/PC-McIDAS system and provided the associated broadcast of research-quality data with university control over the content.

Finally, I wish to thank all the members of our McIDAS Broadcast Evaluation (McBE) committee, Implementation Working Group, and Policy Committee for their efforts. The McBE committee has completed its work, and great benefit has been derived from their collective experiences in the use of PC-McIDAS and the Unidata broadcast. The Implementation Working Group has been immensely helpful in designing the USSDM and, as I'm sure they know, much work remains!

Manager's Report

USSDM

LDM

For over six months now, the LDM has been ingesting current weather data from the Domestic Data Plus (DD+) and Numerical Products (NMC GRIB) services at the UPC and at several test sites using Sun Workstations, Digital Microvaxes, and now IBM RT computers (at Purdue). With considerable help from John Horel and Lloyd Staley at the University of Utah, Jim Cowie at the Naval Postgraduate School, and Harry Edmon and his staff at the University of Washington, the major bugs in the early release of the LDM have been identified and corrected. This software appears to be quite reliable now.

Decoders

Bill Davis (Cooperative Institute for Research in the Atmosphere, Colorado State University) recently delivered decoders for the upper air and surface (UA and SAO) data streams for the DD+ service. UPC's system group is now integrating these decoders, which create netCDF files, into the LDM.

NetCDF

Glenn Davis and the other members of the UPC systems group have developed a network-oriented realization of NASA's Common Data Format concept, described in an article by Lloyd Treinish of NASA Goddard in the 14 July 1987 issue of EOS. The UPC version was written in the C language and employs an emerging standard for transportable data (known as the External Data Representation or XDR). It maintains the functional interface defined by NASA, but the underlying code and file structures will serve in a heterogeneous network environment. We're calling the Unidata version the netCDF, because it supports network access by machines from several vendors running different operating systems.

Cathy Cormack has written a library of Fortran "jacket" routines that enable Fortran applications programs to access the netCDF files. Angel Li at Miami adapted Glenn's UNIX prototype implementation to run on VMS.

There still remains considerable work to move from the prototype to the production version of the netCDF. The production version will include the ability to access the multiple data points with one function call, a facility we call "hyperplane access."

Demonstration Applications

With significant contributions of both software and volunteer manpower contributions from Ross Aiken and Dan Vietor (who work in Ernie Agee's group at Purdue), Bob Green and Tom Yoksas of UPC's User Services staff at the UPC have implemented a number of applications programs that access and display data from files written by the LDM. The applications, which demonstrate access to current LDM weather data, use commercial GKS packages for graphical output. The Purdue suite is running on the UPC Sun workstations, and most of it has also been ported to the Digital GPX Microvax. In their current form, the programs do not access the data in netCDF form, but use the straight text output of the LDM DD+ system. Mike Schmidt (UPC) is adapting a program that presently displays NMC gridded data on the Sun for use on the GPX.

June Workshop

Several weeks ago, announcements were mailed out for a USSDM training workshop 27-29 June. If you did not receive an announcement, contact Lisa Benson at the UPC.

PC-McIDAS

User Support Inquiry Tracking

Two-Way Communications

NCAR Workshop in Synoptic Meteorology Instruction

Workshop on Synoptic Meteorology Instruction

The development of relatively inexpensive computing equipment and the introduction of compatible software systems such as Unidata/PC-McIDAS have made possible the use of computer-driven data acquisition and display systems in courses on synoptic meteorology courses in nearly every university. While such systems should be extremely valuable, it is not immediately obvious just how to incorporate their capabilities into such courses. In addition, synoptic meteorology appears to vary considerably from one university to another, even though completion of a course in synoptic meteorology is considered by some to be necessary before a student can be classified as a meteorologist.

For these reasons, many atmospheric scientists have felt that the time is right for a general discussion on just what should be included in a synoptic meteorology course and how it should be taught. The Advanced Study Program of NCAR, with the aid of UCAR's Unidata Program Center, is organizing a workshop this summer on Synoptic Meteorology Instruction, to be held in Boulder 25-29 July.

The goals of this workshop are to discuss how synoptic meteorology should be taught today's computer capabilities. It will be organized around a series of lectures, roundtable discussions, and hardware demonstrations designed to describe the the latest instruction techniques in synoptic meteorology. Topics will include: the scope of synoptic meteorology, important conceptual and theoretical models used in synoptic meteorology to describe the atmosphere, fundamental analysis techniques, and examples of computer-based and computer-assisted class activities.

IBM is arranging for a network of 12 IBM PS/2 machines to be available for the workshop. In addition, Sun Microsystems is loaning a 3/160 color workstation as well as networking software and hardware. These computers will primarily run the Unidata/PC-MCIDAS and USSDM software. (Some of the USSDM software was developed through an IBM contract with Purdue University. The machines will be available to the participants for demonstrations and for experimenting with ideas generated during the workshop.

Sixteen participants, half from small institutions, the remainder from larger research universities, will lead and record the discussions that take place during the roundtables. In choosing the participants, preference was given to people with experience in teaching synoptic meteorology. In addition, nine invited speakers will present talks intended to initiate the roundtable discussions We expected a few unofficial visitors will also attend but, unfortionately, space is limited.

Because interest in the workshop appears to be extensive and there can only be a limited number of participants, the discussions will be recorded, summarized, and published in a volume that, we hope, will be available in early 1989. If the workshop is successful and interest remains high, a second workshop in synoptic instruction may follow in a year or so. The second workshop would focus on higher level, interactive, data acquisition systems such as the new USSDM system presently being tested by Unidata.

McBE Committee Meeting Summary

The final meeting of the McIDAS Broadcast Evaluation (McBE) committee was held 24-25 February 1988. McBE chairperson Rollie Hauser hosted the meeting at California State University at Chico. All members except a representative from SUNY-Albany were present; I represented the Unidata ProgramCenter (UPC). The agenda included: (1) discussion of site experiences with the new software release (open keyboard mode); (2) use of the system at each site; (3) a UPC User Support update (including a summary of the Training Workshop); (4) the place of Unidata/PC-McIDAS in a teaching environment; and (5) a discussion of the McBE final report due this spring.

All members were pleased with the new software and data stream in use since the November 1987 meeting. Their flexibility and product mix made the system much more useful for classrooms. Two sites have had a higher-than-normal occurrence of the system locking up during the day. We hope that the buffer box replacements will help, but sites should check the Zephyr data path for possible problems.

Most members are interested in expanding their systems to include more workstations in a networked environment. Networking seems to be a very important issue with all Unidata/PC-McIDAS users, both McBE and the new users. Sites are testing other hardware configurations and other brands of equipment with varying results. As information on these systems is reported to UPC, we will pass it on to you.

I discussed the "buddy" assignments and the user support procedures presented at the training workshop. I also made copies of the Unidata/PC-McIDAS Manual available to the members.

Tim Spangler (Northern Illinois University) led a discussion of the advantages of the Unidata/PC-McIDAS workstation in a classroom setting. Features and capabilities mentioned were: (1) satellite data and images in near real time; (2) animation of images and graphics; (3) graphic overlays on satellite images; (4) data analysis in time series format; (5) ability to handle case study data sets; and (6) display of NMC model output. Bob Gall (University of Arizona) suggested that two main ways of using the system: (1) in demonstration mode for classroom setting, which requires extended video capabilities (connection to projection television or conversion of the EGA signal to the NTSC video standard for feeding campus video systems in lecture halls); and (2) in interactive mode, for hands-on laboratory use by individuals (map discussions, lab assignments, and research work where multiple, networked workstations are an advantage). Spangler led a session exercise in which a synoptic meteorological situation was presented, and the group suggested how the PC-McIDAS system data and commands could most effectively illustrate the life cycle of the situation.

From the discussions, the committee compiled a list of areas requiring additional investigation to better serve the university needs. These are: (1) fool-proofing the system for more stable operation in a student environment; (2) a simple method for making hardcopy of graphics for student handouts and exercises; (3) an archive of case-study data sets of unique weather situations for classroom playback (a uniform, Unidata file-naming convention would allow exchange of data sets within community); (4) cheaper equipment configurations to make more workstations affordable; and (5) issue of portability of data stream to other hardware, of software to other systems or to PC-McIDAS, and of the video signal to campus systems. The last area is of most concern at this time.

By Friday, there was a major breakthrough in the problem of getting hardcopy of the PC-McIDAS screen graphics. Tom Yoksas (UPC) had worked with John Militzer (NCAR) with some screen dump software and both standard and color dot matrix printers. They designed procedure to get hardcopies of the PC-McIDAS graphics, but not within the PC-McIDAS environment. The hardcopies were sent to the meeting site, and the members were very impressed by the results. The procedures will be documented and made available to the Unidata community by User Support.

USSDM Workshop Scheduled

The demonstration functions for this initial release include: decoders for surface and upper-air data from the Domestic Data Plus (DD+) circuit; display of decoded data as simple tabular text; display of station model data on a basemap; contouring DD+ grid data gridded by objective analysis and NMC binary gridded model output; upper-air kinematic analysis and stability indices from DD+; two-dimensional streamline analysis from DD+; meteorological calculations; and deriving local scalars for surface and upper-air observations.

An announcement of the workshop and an application form was mailed separately to all subscribers of this newsletter.

New PC-McIDAS Release

The SSEC has also announced that it will be altering the frequency and content of test patterns on the Unidata/Wisconsin data channel. The pattern will now be sent every ten minutes with an extra pattern being transmitted during the hourly RESET sequence. This change will be transparent to the users.

Unidata in the Literature

The April issue of the Bulletin of the American Meteorological Society contained three articles on Unidata or its systems. Lynn Sherretz and Dave Fulker presented an overview of the Unidata program, and supplied the graphics for the Bulletin's cover. (Dave is Unidata's director.) Two articles under educational affairs in the same issue focused on programs using Unidata services.

The Unidata Program Center would appreciate information on all mentions of its system or programs in the professional literature.