Table of Contents

• Director's Report
• Manager's Report
• Revised Plan for Implementing the UNIX/VMS Unidata System
• MSDOS, UNIX, or VMS? The Choice is Yours
• Connecting Scientific Workstations to Supercomputers
• Unidata Software License Agreements

Director's Report

I have experienced quite a range of feelings over the last three months, as Unidata met a mixture of successes and setbacks. The excitement of seeing our six test sites demonstrate the usability of PC-McIDAS and the power of the Unidata/Wisconsin broadcast was offset by the disappointment of a breakdown in collaboration with the providers of GEMPAK. These could not be resolved quickly, and we found it necessary to develop a new strategy. Associated problems, especially deciding on a replacement for the critical data-structuring software, were compensated (emotionally, that is) by the successful testing of our prototype local data manager (LDM). The LDM receives and organizes input simultaneously from the National Weather Service Domestic Data (Plus) and the National Meteorological Center Gridded Data circuits available through Zephyr Weather Information Services.

Our earlier plans called for the full integration of GEMPAK version 4 data-structuring software into Unidata's LDM system. Our revised plans utilize a more general data-access methodology, based on the common data format (CDF). The versatility of the CDF interface will permit the LDM to support a variety of meteorological analysis and display packages adapted and contributed by members of the Unidata community; it will also allow users to access various National Aeronautics and Space Administration (NASA) databases.

I'm sure that some of you are disappointed by news of our difficulties regarding GEMPAK. The package is well engineered and provides a comprehensive and flexible set of capabilities for working with surface, upper air, and gridded data sets. Our plans for its use, especially for the decoders and data-access routines, were based on on the assumption that portions of GEMPAK would be adapted collaboratively (and on a tight schedule) to meet the Unidata requirements. Unfortunately,the problems developed just as our need for access to the critical components peaked. To continue our progress, we devised a new plan that eliminates our dependency on GEMPAK and permits us to proceed with the development of a production-quality LDM system and an integrated set of (university-contributed) analysis and display capabilities.

In October, the Unidata Implementation Working Group (IWG) endorsed this revised plan, which was subsequently approved by the Unidata Policy Committee. The outcome of the IWG was most gratifying, as members of the Unidata community came forward with valuable suggestions and generous offers of assistance. I wish to extend my personal thanks to Ross Aiken and Dan Vietor of Purdue, Niel Allen and Bill Davis of Colorado State, Otis Brown of Miami, Harry Edmon of Washington, George Huffman of Maryland, Art Persons of Penn State, David Raymond of New Mexico Tech, and their respective institutions for the support and guidance they provided.

Technically, we are in a strong position for the future of Unidata. Our revised plan incorporates a data-access methodology suitable for image data as well as for conventional surface and sounding data. We believe the method is likely to become a NASA standard, so the new approach will increase compatibility between Unidata and a number of major data holdings at NASA, including those of the Pilot Climate Data System, the Pilot Land Data System, and the collections used in the Coordinated Data Analysis Workshops in support of solar-terrestrial physics. The willingness of our university collaborators to adapt applications software to the new CDF interfaces, combined with the flexibility of those interfaces, suggests to me that our final result will provide comprehensive capabilities and a high degree of user programmability.

Thus we have refocused our UNIX and VMS efforts toward managing and organizing data. We are emphasizing the flexibility and programmability of the system, and we intend to encourage participants to interface new and existing software to the Unidata LDM system. Users will employ a collection of Unidata standard FORTRAN- and C-callable subroutines that will work across the Unidata standard networks. We think users will find these routines easy to use and effective for accessing the varied datasets that can be captured by the Unidata LDM system.

Furthermore, our new plan does not preclude the eventual use of GEMPAK in Unidata systems. We believe GEMPAK can be successfully adapted to work with our revised data-access method. An invitation has been extended to NASA offering to include GEMPAK in the Unidata library on the same basis as other software packages contributed by the community -- that it comform to the CDF data-access interfaces. In any case, we have planned to integrate a varied collection of applications software already developed by universities, encompassing a significant range of scientific needs.

I conclude this note with a suggestion that you examine the system configuration specifications provided in this issue. The Unidata version of McIDAS will be released for general use next February, so those of you with MSDOS systems will soon have information on how to install it. Next semester you can have pseudo-color satellite imagery and other computer-based data analysis and display capabilities at the fingertips of your students and yourselves, at very modest prices. I hope you find this an exciting prospect, and I look forward to hearing of your interest.

Manager's Report

As you have learned by now, GEMPAK Version 4 (meteorological display software developed by the National Aeronautics and Space Administration [NASA]) is not presently available for integration into the UNIX/VMS Unidata workstations. This fact has had a significant impact on the development of Unidata's local data management (LDM) system. Decoders for the upper air (UA) and surface airways observations (SAO), along with data-access libraries for the Version 4 data structures, were part of the package that was to come with GEMPAK. Instead, Colorado State University has begun work on SAO and UA decoders for the domestic data (DD+) stream supplied by Zephyr Weather Information Services, and the Unidata Coordinating Office (UCO) is working out arrangements for an alternate set of data structures and access libraries. This is based on a method described by Lloyd Treinish and Michael Gough (NASA) in a paper appearing in the 14 July issue of Eos.

The Unidata Coordinating Office and its Implemetation Working Group are identifying and planning to integrate a set of analysis and display capabilities to complement the data-access utilities. So far, the plan relies heavily on the contributions and existing programs of Unidata participants.

Graphics

PC-McIDAS

Two test sites (Northern Illinois University and our office) have implemented a limited form of network access. It is not yet possible to run PC-McIDAS and the file server software at the same time. Instead, at both sites, a signal in the broadcast stream triggers an interrupt of the ingest system and boots the system in a mode that enables transfer of the data files to a file server on the network. The system is then automatically rebooted, so that the data ingest can continue. (Buffer boxes on the incoming data line prevent loss of data during this process.) Other systems on the network can then access the data from the file server for analysis and display.

The file server in the Coordinating Office is actually on a UNIX-based Sun workstation, so copying the PC-McIDAS data files to the server has the beneficial side-effect of making them available for analysis and display on other UNIX and VMS workstations on the network.

The first PC-McIDAS workshop and the general release of software is scheduled for the latter part of February 1988. A detailed announcement will be mailed in December to everyone receiving this newsletter.

Local Data Management for UNIX and VMS Systems

The Implementation Working Group agreed on the upgrades needed to bring the current prototype system up to production status. Di_Sys, Ltd., the subcontractor who developed portions of the LDM, is working with UCO's systems group to implement the production LDM in time for alpha-testing next month.

User Support

Unidata Exhibit

Staff Changes

Revised Plan for Implementing the UNIX/VMS Unidata System.

• Building the production version of the Unidata local data management (LDM) system. The prototype LDM is now operational. Building the production version involves working with Di_Sys, Ltd. to incorporate changes specified by the IWG/LDM subgroup and approved by the full IWG. This production version will be running in the Unidata Project Office by mid-November.

• Testing the production LDM. This begins in mid-November at a few test sites where suitable hardware and expertise are available. No Applications software will be provided to these sites.

• Incorporating surface airways observation (SAO) and upper air (UA) decoders. These were to have been supplied by NASA/Goddard Space Flight Center (GSFC). Now they are being written by Colorado State University. They, too, will be delivered to us by mid-Novemberand will be incorporated into the production LDM.

• Integrating a data-access interface. Unidata, with the guidance of the IWG and the endorsement of the policy committee, has adopted the NASA GSFC (GSFC) common data format (CDF) subroutine as the mechanism for writing and reading data files. LDM decoders will use these routines to write out files, and applications will use them to access the data in those files. This integration will involve steps:
  1. An agreement has to be struck with the appropriate parties at NASA to ensure that questions regarding access to the code, licensing, and support are understood by both parties. Drafts of such an agreement have already been written and approved in principle by key individuals at Goddard and Unidata.

  2. The current version of the CDF (which is written in FORTRAN for VMS systems) will be implemented on the Unidata's VMS MicroVAX GPX II after Unidata staff member Glenn Davis visits GSFC and familiarizes himself with the overall CDF system and the code.

  3. Assuming the first steps go as planned, Unidata will be testing and evaluating the working CDF system in early December.

  4. In parallel with the implementation of the current CDF, we will begin to work on defining a C-language binding and implementation of the CDF for UNIX as well as VMS systems. This revised CDF is being called the "network CDF."

  5. Implementation of the network CDF on UNIX and VMS is scheduled for early February. This, of course, depends on the earlier agreements and implementations being in place as scheduled.

  6. Starting in February we will be testing the network CDF library first here at the Coordinating Office then at test sites. The tests are scheduled for completion by the end of April 1988.

• Implementing interactive application programs for data analysis. Application programs are to be implemented in such a way that they access the CDF files created by the LDM. These programs will be adapted from those already in use at various universities in the Unidata community. Here, too, several steps are involved:

  1. A system for identifying the needed applications and for evaluating potential candidates is already in place and has been applied to working applications programs at several sites. We will continue to use the same scheme, and potential candidates will be identified and evaluated in time for the mid-December IWG meeting. The IWG can then decide which applications should be pursued further.

  2. Unidata will select and begin implementing a few basic applications so that the display utilities can be tested with the LDM/CDF system during March and April 1988.

• Choosing a uniform user interface. The user interface for the applications ultimately chosen by Unidata has not yet been selected. Its integration will be done when it can be combined with the satellite image display facilities as well as those for conventional data. Candidates under consideration for the uniform user interface include the NASA TAE (transportable applications executive) system and the Miami DSP (for image display) system.

MSDOS, UNIX, or VMS? The Choice is Yours

Many potential Unidata sites have decided which system to implement. For those who are still undecided, this article summarizes the fundamental differences between the two options and how they will evolve over the near term. (It should be noted that one of Unidata' long-range goals is to develop a system that allows both types of operating systems to exist on the same local area network [LAN] and to even share a common LDM. This goal has been only pertially realized at present.)

MS-DOS Systems

Over the next few months, the current menu-only interface will be augmented by an option enabling users to access individual programs themselves and to exercise more control over the functions of the system. In this command mode, users will be able to set up displays other than those offered in the menus.

During the next year, the system will be transported to the Microsoft Corporation's OS/2 operating system. This should remove current restrictions resulting from McIDAS's limited program size and its inability to run programs simultaneously.

At present, a limitation affects installations where more than one workstation is involved. Using PC-McIDAS on a LAN can be done but it requires some effort on the user's part to enable the machine ingesting the data to copy them to other machines on the network An alternative is to split the incoming data feed and send data directly to individual workstations.

For sites that wish to develop their own applications software, Purdue University has tailored the NCAR graphics package for interactive use on MSDOS systems. Ultimately, users will have access to portions of the source code for the PC-McIDAS applications programs and for the library interfaces to the core of the system. Users will then be able to tailor applications to their own needs.

UNIX/VMS Systems

In terms of data availability, the PC-McIDAS option can access only the preselected Unidata/PC-McIDAS data stream while the UNIX/VMS option allows access to the full range of NMC grid data and to "raw" conventional data. In terms of existing applications, however, PC-McIDAS is functioning while the availability of applications programs for UNIX/VMS are not completely determined at this writing; even the LDM system will not be available until the second quarter of 1988.

Finally, the UNIX/VMS option involves systems that are sophisticated and flexible; systems that can be used for other purposes as well as for Unidata. This means, however, that users will have to make decisions regarding hardware, vendors, and operating systems. In general, the UNIX/ VMS option requires more computing expertise on the part of the users.

Summary

Connecting Scientific Workstations to Supercomputers

by Ernest Agee, Rose Aiken, and Maria Zivkovic, Purdue University

The microcomputer has rapidly become a useful tool for scientific researchers where a high degree of interaction is required. Program development, graphical data display, and publication of research results are tasks best accomplished at one's office desktop microcomputer. However, the information on one's desk is not widely useful unless it is accessible when and where it is needed, nor is the workstation powerful enough to hold large data archives or to perform large model runs. These require communication networks.

Connecting scientific workstations to each other and to supercomputers is an ambitious and complex communication task, but one which is worth striving for because of the flexibility it offers the atmospheric research community. This article illustrates one method of achieving this, consisting of multiple interconnected package switching networks that function, via the CYPRESS network, like a single large network.

Some Definitions

The objective of an internet is to transfer data between processes running on different computer systems Different levels of activity are necessary to achieve this transfer and it is useful to conceptualize the mechanisms involved in terms of layers. The standardization effort of the International Standards Organization called the Open Systems Interconnection (OSI) model fosters this view.

The OSI model contains defined protocols for messages and their exchange, hierarchical structure of tasks, functional grouping of tasks, defined boundaries between tasks (interfaces), encapsulation of data with headers and trailers, and defined communication between peer tasks as well as tasks on different levels of the hierarchy.

The seven layers and their functions in the OSI model are:

1. Physical. Specifies the physical medium over which the bits of data are being transmitted and thereby involves the mechanical/electrical concerns of the hardware.
2. Data Link. Are the mechanisms needed to form the sequence of bits with headers and trailers to form data frames for transmission between parties.
3. Network. Involves the assembly of frames into packets to provide routing among nodes of a network; sequencing if appropriate also occurs at this level.
4. Transport. At this level a definition of a set of end-to-end services between various processes on various hosts along the network is present.
5. Sessions. Manages dialogue between users as well as sets up and terminates communication in the network; somewhat analogous to network log-on.
6. Presentation. Deals with matters of representation of data elements, code conversions, format translations, data encryption, and compression; is entirely a software function.
7. Application. An end-to-end, application-specific layer, which includes applications like remote log-in, job submission, file transfer, electronic mail, and terminal support.

The OSI model provides defined and differentiated methods of intercommunications, with repeaters, bridges, and gateways. Repeaters extend local-area networks (LANs) past their normal physical boundaries and operate at the bit or physical layer. A bridge permits two LANs to function independently and so to operate on the data link layer. Gateways operate at the packet or message level that correspond to the network or transport layers so that dissimilar LANs can communicate.

The vast diversity of LAN technology and user demand for interoperability has led to this standardization effort. Reference to this model and its terms keeps us from being overwhelmed by semantic differences.

The CYPRESS Experiment

The CYPRESS Project, under the direction of Douglas Comer (Computer Science, Purdue University) has sought to develop a network of low-cost access via a capillary system to the arterial network of ARPAnet and NSFnet. The computers involved all use the standard TCP/IP protocol for reliable data stream support. IP (for Internet Protocol) defines the format of the packets passing through the internet, including the details of the source and destination addresses, packets size, and the order of fields within the packet. TCP (Transmission Control Protocol) defines how computers cooperate to ensure that data have passed correctly and reliably across the internet, including details of error detection and retransmission, buffering, and demultiplexing.

The CYPRESS network consists of a set of packet switches called implets, connected together by dedicated, leased, data circuits. Implets can exchange packets with adjacent implets by sending them over interconnected leased lines. Each implet functions as a store-and-forward packet switch, accepting incoming packets and routing them on toward their destination, based on a destination address found in each packet.

One implet resides at each subscriber's site, where it attaches to the user's LAN. One of the most important ideas in CYPRESS is that the implets communicate with the subscribers' machines over a LAN like an Ethernet, making the system loosely coupled; the subscriber's machines use the implet like an IP gateway connected to the internet.

Work on the CYPRESS project has been in progress at Purdue since 1985, and it has yielded an optimized set to deliver IP packets which makes efficient use of slow line-speed baudwidth (9.6 kb) and still preserves all the functionality of IP.

An IBM PC/AT at Purdue provides the beginning device and it ends at the IBM 4381 that serves as a front-end processor for NCAR's CRAY X-MP. Although it appears that the PC is a screen on the IBM, there is in fact a long line of hosts between them. A typical session travels through the following network layers:

1. The IBM PC in the Department of Earth and Atmospheric Sciences at Purdue sends information to the CYPRESS hub machine via an X-box implet attached to the department's Novell Netware-based LAN (which is shared among over 20 workstations).
2. From the CYPRESS hub, information is put onto ARPAnet via a gateway machine in the Department of Computer Science.
3. Once on the ARPAnet, dynamic routing across any number of nodes is possible. Most likely, this information will be passed onto a route ending at Carnegie Mellon University, where a gateway interfaces with the NSFnet backbone.
4. The information travels to NCAR on NSFnet, where it is pulled off and placed on a path to the Scientific Computing Division's IBM 4381.
5. Specialized facilities on the IBM 4381 allow remote job entry to the CRAY, which is connected to the IBM via Hyperchannel.

The Benefits of Cooperative Processing

Unidata has as one of its goals the development of facilities for connecting to nationwide communication systems. Such a connection would provide universities with a means of sharing programs, data, and research among the user community, accessing national archives of historical data, and accessing major national computations resources such as NCAR's supercomputers. Using our experience with CYPRESS, we are helping Unidata evaluate the ability of the CYPRESS techology to achieve these goals.

Becoming a CYPRESS Participant

Unidata Software License Agreements

Below is a statement adopted by the Unidata Policy Committee on the subject of software distribution and protection. As a result of this policy, both providers and users of Unidata software will be asked to sign software license agreements. Alhough this may seem like a stuffy, legalistic action, the underlying intent is to pave the way for community software exchange while protecting the rights of the originators. The final license agreements are not yet complete, but if you have questions or would like copies early before you are ready to participate, please contact the Unidata Program Office.

Unidata Software Distribution and Protection Policy

The success of Unidata is built upon community participation. This requires the free and open exchange of software amongst the Unidata Program Office, its university and government agency collaborators, and the university end users. There is also a need to protect the rights of software developers. To this end, the Unidata Policy Committee recommends the following guiding principles to UCAR, to collaborating government agencies, and to other potential Unidata collaborators, for guidance in their interactions with the Unidata program.

1. Broadest possible utilization of Unidata software by the university community.
2. Dissemination of Unidata software to university users at minimum cost.
3. Protection of intellectual property rights and interests of software developers and/or their employers first, and of UCAR, second.
4. Participation in the Unidata program by private industry, possibly as users as well as for collaboration and provision of services, and possibly as a source of revenue, consistent with the above principles.

Furthermore, the policy committee asks that UCAR establish, as soon as possible, copyright or licensing procedures for Unidata software which adhere to the above principles.

(End of Policy)

UCAR has developed drafts of licenses intended to implement this policy. These are currently being revised in response to suggestions by the Unidata Policy Committee at its October meeting.