Software engineering is often the poor cousin of science when scientific programs are written. Scientists typically learn enough of a programming language such as FORTRAN to be able to efficiently code algorithms for their everyday work. Longer programming tasks are relegated to younger and more junior scientists, and even more often, to students. It is often the case that no one who touches the code has studied computer science or software engineering for its own sake. Even those who might be called programmers are typically converted scientists, whose interest in the job comes from their background in the field.

At the same time, there is a good deal of code reuse in scientific programming. There is a strong tradition of sharing code among colleagues, and of making programs freely available. The science community is highly motivated by peer recognition for individual contributions to community knowledge and resources. It shares the same social characteristics as the "free software" and "hacker" communities of UNIX/linux, FSF, and similar free software communities.

Unfortunately, the lack of sound software engineering practice makes much of the contributed software unreliable and difficult to reuse. To some extent this reflects the immaturity of software engineering itself. Another contributing cause is the predominance of FORTRAN, whose design reflects its early origins. While there has been some movement toward object oriented software in the scientific community, this movement has been slow and sporadic. Also, FORTRAN’s very primitiveness makes it easy for compilers to optimize, and so efficiency concerns have hampered movement toward more modern languages.

The current interest in component software provides a fresh opportunity to examine the possibilities of code reuse in scientific programming. The availability of cheap and powerful desktop computers, driven by the business and entertainment markets, makes efficiency less of a concern for some kinds of scientific programs, such as data analysis and visualization. In principle, even FORTRAN code can be packaged as component software for reuse.

This paper examines some of the issues involved in applying component software development to scientific data analysis and visualization. Part 2 gives a brief summary of component software concepts so that this paper may be more broadly accessible. Part 3 surveys related work. In Part 4, I lay out some context and goals for component software development within meteorological applications. Part 5 reports on a simple prototype, and Part 6 describes conclusions and future work. In the appendices are more complete designs and mathematical descriptions of meteorological data that may be of interest to domain specialists.

Software components are units of software designed to interact with other independently developed components, and to be assembled by third parties into applications. Software component engineering focuses on the packaging of software into independent units to allow maximum reusability. (For an in-depth discussion of component software, see Szyperski97).

Component-based software development (CBD) is an evolution of object oriented software development (OOD). While both share the goal of software reusability, OOD is an implementation methodology, while CBD is an interface methodology. In CBD the emphasis is on standardizing the interfaces between components, with no restrictions on how the implementation is accomplished. CBD therefore is closely related to module design in separating interface and implementation..

In OOD, code reuse is accomplished through inheritance of implementation code. While OO languages also generally allow separation of implementation and interface, in practice the desire for code reuse often complicates data typing, and inheritance breaks data encapsulation (Snyder86). Partly for these reasons and partly due to the extra level of effort to prevent unnecessary dependencies, class hierarchies tend to be reused only within an application. Of course, class libraries as well as procedural libraries designed to be incorporated into larger applications are successful examples of software reuse.

Components require a deployment environment that Szyperski calls a component world or component environment. The services that a component needs from its environment are called its context dependencies. The mechanisms by which components interact are sometimes called wiring standards, and are one of the most important specifications of a component environment. There are currently three major component environments: The Object Management Group’s (OMG) CORBA, Microsoft’s COM, and Sun’s JavaBeans.

CORBA is an interface standard for distributed objects which may be implemented in different languages and running on different platforms. An object is defined with an Interface Description Language (IDL), and language and platform specific bindings must be created by CORBA vendors.

COM is a binary standard that defines interfaces between components. It grew out of Microsoft’s highly successful Visual Basic controls. A COM component can be implemented in any language, although is most often created in Visual Basic or C++, since Microsoft provides tools for those languages. COM is supported for MS Windows. MacOS, and by third parties for other platforms. COM supports interface inheritance but not implementation inheritance, and allows components to be nested inside of one another. DCOM is an extension of COM to distributed components. COM+ is an object-oriented version of COM for components within a single process.

JavaBeans is a set of conventions for Java to support component-based software development. The support for these conventions is embedded in the Java language and supporting class libraries, so JavaBeans is a Java-only component framework. However, using the Java Native Interface API, Java classes can wrap components built in other languages, can become COM components, and can be CORBA components as well.

A component describes itself through its interface, which can be thought of as a contract with whoever uses it. The contract has two parts: the syntax specifies the data types of the interface method’s arguments, which can be enforced automatically at compile or run time. The contract semantics describe the behavior of the component, and can only be understood by humans, and so typically is represented by the documentation that the component developer provides. Because semantics are always context specific, the standardization of component behavior is necessarily specific to an application domain, and are called domain standards. The OMG has created several "domain task forces" to create domain standards for business, manufacturing, electronic commerce, telecommunications, finance, and medicine. Both the Java and OLE communities also have domain specific efforts.

The combination of domain standards and supporting implementation classes and components that facilitate independent component development is called a component application framework, the most familiar example being GUI application builders. (Szyperski uses the term in a more restrictive sense, with the compound document framework OpenDoc as a standard example. But my definition seems to be more widely used in the industry). A component framework must specify both wiring and domain standards for the components it provides, that is, both syntactic and semantic descriptions of its interfaces. In practice, a component framework is an implementation of a domain standard and so both simultaneously evolve out of each other.

The idea of an application development framework as a platform for building domain-specific applications is certainly not new with component software. Scientific visualization systems abound: since most scientific work is government funded, many tools and other software are placed in the public domain by scientific workers who have developed the software in the course of their work. As noted before, this work is of variable quality, and typically is the effort of one person or a small group, though perhaps developed over many years. In this section I briefly survey a few of the available freeware systems for building scientific analysis and visualization applications.

Sluice (Sluice97) is a Java-based framework for constructing collaborative scientific visualization applications. It uses a dataflow network (like AVS and DataExplorer). Data is presented uniformly as two-dimensional "tables" of arbitrary Java objects. Processing modules can alter these tables, and visualizing modules read and display based on them. The Sluice framework include interface specification, design patterns (method naming conventions), and a support library. They use Java introspection methods to connect modules, and apparently are following JavaBean conventions, and plan on "exploring new ways of exploiting Sluice's compliance with JavaBeans interface definitions and design patterns."

Vis5D is a scientific visualization package and VisAD allows scientists to interactively "steer" and visualize their computations, both from Space Science and Engineering Center (SSEC) of the University of Wisconsin-Madison (Hibbard94). Unidata is working closely with SSEC to develop and share class libraries for a Java rewrite of VisAD (VisAD97). VisAD has a sophisticated mathematical data model at its core, providing class methods to perform algebraic operations and functional transformations. The visualization classes use the Java 3D API for generating displays. While VisAD itself does not use JavaBean components, it provides an implementation hierarchy on top of which domain specific applications can be developed using components.

The Linear System Analyzer (LSA) from Indiana University is a component framework for solving linear systems of equations (LSA98). It provides components that encapsulate the, as well as a GUI for composing the components. The developers of LSA cite the size and diversity of data objects in scientific systems, as well as economic factors to explain the slow penetration of component software. They conclude that components must be accessible to procedural programmers, that peer-to-peer communication is needed between component frameworks themselves as well as components, and that several communication subsystems are needed, "ranging from graphical cues for the user interface to high-performance communications paths between components".

The Caltech Infospheres Project researches compositional systems, which are systems built from interacting components that support peer-to-peer communication among persistent multithreaded distributed objects. Their research concentrates on "developing reliable distributed applications by composing existing and newly created software components in structured ways". (InfoSpheres96)

Yorick is a programming language for scientific visualization and analysis, developed at Lawrence Livermore National Laboratory by David Munro. It is an interpreted language with powerful array syntax, complex numbers and built-in mathematical functions. Its visualization tools are highly integrated into the language, and make data analysis and visualization seamlessly integrated, though at the cost of relatively steep learning curve.

Other mathematical models for scientific data include (Butler89). For a more comprehensive survey of visualization packages, see http://science.nas.nasa.gov/Groups/VisTech/other/save/visWeblets.html.1

I am interested in exploring component-based software development in the context of the academic and research meteorological scientific community. Specifically, I work for Unidata, which is a National Science Foundation (NSF) funded group whose purpose is to assist universities to acquire and use meteorological data.

Unidata has successfully deployed a store and forward data distribution system called the Internet Data Distribution (IDD). The data includes the National Weather Service (NWS) surface and upper-air observing stations, National Center for Environmental Prediction (NCEP) assimilation and forecast models output, NASA satellite data, as well as private companies such as WSI national weather and radar composites, and Global Atmospherics’ National Lightning Detection Network (NLDN).

Unidata also supports several data display systems, including McIDAS, Wxp, and GEMPAK. All are legacy systems written by other groups. Since Unidata supports a diverse community of universities and research groups, the target platforms for these systems is highly heterogeneous. Maintaining and porting these legacy systems to new hardware and software platforms requires significant resources. Recently, Unidata has decided to use Java for new development due to its platform independence, and to gradually replace these legacy display systems.

Unidata, by virtue of its position as data provider to its member universities, is in the position to define and promulgate new standards for meteorological data dissemination and display. Furthermore, Unidata is primarily a software-engineering group with extensive experience in the domain of meteorological and scientific data. It is therefore rather uniquely positioned to provide software engineering leadership to the academic and research community, which typically lacks such expertise.

Finally, one of Unidata’s missions is to encourage member universities to "cooperate, collaborate, and share a variety of resources", including data and software, in line with William Wulf’s proposal for a national "collaboratory" (Wulf89). While the IDD is a practical realization of data sharing, the possibility of software sharing has remained mostly unrealized, primarily due to the monolithic nature of application development. Component software offers a new way to imagine the shape of possible software sharing.

The idea then, is to develop a component framework for meteorological data analysis and visualization, that enables code sharing and cooperation among participating institutions. The goals for this framework are:

1. Data sources and data consumers must be able to be developed independently. Data interfaces must be as general as possible to allow data consumers to work with multiple data sources. This means that if there are n data sources and m data consumers, there should only be on the order only n + m objects, rather than n x m objects, in the application.
2. Data consumer classes should be independent of other data consumer classes. For example, a data display component developed for one application should be reusable in another application. This is challenging when you are trying to create seamless GUI applications.
3. Application GUIs should be able to be assembled in a visual GUI builder tool. Our goal is thus to allow independently developed data source and data consumer components to be easily assembled into an application.

Overview

In order to evaluate the use of components in scientific data analysis and visualization systems, I implemented a very simple data visualizing system, called jmet, using JavaBean components. There were three types of components used in jmet:

1. Data Provider. These are components that read specific types of scientific datasets or data sources, and provide standardized access to them. I implemented a DataSource class that encapsulated access to a global atmospheric chemistry model dataset stored in a netcdf file.
2. Data Consumer. This kind of component takes data from a Data Provider and manipulates or visualizes it. I implemented two components: class HGridView displays a horizontal 2D slice of data with a global map overlay, using "false color" rendering; and class PlotView shows an x-y plot of a vertical profile of data.
3. GUI components. Class jmetChoice is an extension of java.awt.Choice that makes hooking a Choice component up to a DataSource easier. Other GUI components such as Labels were used from java.awt without modification.

I used a commercial Java integrated development environment (IDE) for this prototype in order to understand the current state of the art in visual GUI builders for JavaBeans. I considered the four IDEs that have the most market share: IBM’s Visual Age, Symantec’s Visual Cafe for Java, Borland’s JBuilder, and Microsoft’s J++. Visual Age (1.0) does not yet support inner classes, which is necessary to use the Netcdf Java libraries, so I could not use it for this project. I did use Visual Cafe (at first version 2.1, then an upgrade to version 2.5). JBuilder (1.01) and J++ (6.0) both arrived too late to evaluate for this project.

All source files for the project were created using Visual Cafe. The components were placed into jar files and added to the "Component Library" so that the GUI could be built using Visual Cafe’s drag-and-drop application builder. I also tested the generated application from the java command line. I later tested all of the beans in the beanbox, but did not generate a working application from it.

Implicit in the separation of Data Provider and Data Consumer are abstract data types (ADT) that describe the data that needs to be shared between the two. Interface GridDataset is a very simple version of an ADT that describes a type of data called a Grid. All information and methods on data objects are described in this interface; the Data Provider must provide an implementation of it. GridDataset is thus the application programmer’s interface (API) to the Grid data type.

I used a modified Model/View/Controller design pattern as described below. Class GridController implements the "controller" part of the MVC. This class acts as the "glue" for the application, because it knows about all of the other classes (except the DataSource class, which is accessed only through the GridDataset interface), and adds the interactions between them. None of the other components know about any other, so that they can be created and used independently of each other. In the section on "automatic wiring" below, I add some thoughts on assembling applications and adapter classes.

In the standard Model/View/Controller (MVC) design pattern, the model encapsulates the data and state information, the view displays the data to the user, and the controller receives user input and translates it into requests to the model or view (Buchman96). There may be multiple views, typically each with its own controller, for each model. Changes to the model state trigger events to the views and/or controllers which then redraw themselves. MVC, first used in the Smalltalk development environment, is used extensively in GUI libraries, including the Microsoft Foundation Class (MFC) library and the new Java "swing" library.

Scientific datasets are notorious for their size; typically they grow to the size of available disk and memory resources. Therefore efficient data subsetting and minimal data movement is a strong requirement of all scientific analysis packages. Data is usually made immutable to allow efficient caching strategies. Meteorological applications essentially consist of navigation through complex datasets, then selection, operation, and display of data subsets. The state of the navigation and the selection must be maintained, along with triggering the appropriate data fetching, operation and display when the selection is completed.

For jmet, the model component is provided by the Data Provider objects. Since these are immutable, they cannot contain state information. I chose to put the state information in the controller object, and to have a single controller object for both views. The view component was provided by the Data Consumer classes. State change events are generated both by GUI components and by the view components themselves (for example in response to mouse clicks within the view’s window). These are sent to the controller, which keeps track of state and triggers data updating and redraws (see Figure 1)

Figure 1.

One advantage to this design is that the controller object is interposed between the Data Provider and Data Consumer classes. In fact, the controller is the only class that knows about the GridDataset interface, not the Data Consumers which are free to define their own interface. GridController thus becomes a (rather complex) adapter class between the Data Provider and Data Consumer, in addition to handling GUI events. While this may facilitate independent production of the Data Provider and Data Consumer classes, and this works well in the case of this very simple prototype, it is likely that more complex applications will require a different design pattern.

If so, it might be useful to create a model component class which becomes a proxy for the Data Provider object. Navigation state is kept in that model component, and so conform to the standard MVC design pattern. The controller class still manages the various events, but may be somewhat simpler because some of the Data Provider behavior is now in the model object.

What’s the component and what’s the application?

In this simple prototype, the Data Consumer components (views) are created without a controller component specific to it. In some ways this should make these Data Consumer components more reusable, since an application developer can add his/her own look and feel and achieve consistency across all components.

The tradeoff is that as components have more complex behavior, it becomes more work to wire up appropriate GUIs. A view that has much of the necessary GUI already embedded in it may be more desirable, and so more reusable. (A view with embedded GUI components may want its own controller object, as in the classic MVC pattern; a common modification is to combine the view and controller into one object)

To understand this tradeoff within the prototype application, consider the following state requirements. The HGridView wants a 2D (x,y) array of data to display. Since it is hooked up to a 5D dataset (field, time, z, y, x), the application must provide a selection GUI for the other 3 dimensions, and maintain the state of those selections. PlotView wants a 1D array (z) of data (with coordinates for the data points). This is selected by clicking in the HGridView canvas, which generates an event containing the (x,y) selection. The class GridController, which maintains the HGridView state (field, time, z), then synthesizes the selection (field, time, x, y), obtains the 1D array and calls PlotView to draw it.

For this prototype, placing the GUI in the application rather than in one or more of the views, makes the management of state information easier. It also allows different views to share the same state, and thus enables consistent "seamless" integration of the components into the application. From this point of view, it is the application developer’s job to provide a single GUI, along with the interconnections between the components. For developing "sovereign" applications, components likely need to relegate most or all of the GUI to the application builder.

There may be another class of applications, more akin to compound documents, which might embed "heavyweight" components that contain all or most of their own GUI. These might be useful for education or web-based applications.

Visual Wiring Tools and Adapter Classes

If a programmer is familiar with the API for all of the classes in the application, generating the interactions between the various objects is easier in code than in a visual "wiring" tool. For this reason I first wired the components together in code in the GridController object. Since there are three "choice" components, each needing to notify the GridController when a choice was made, I added three inner classes, and passed the choice objects into the GridController constructor where I wired them:

times.addSelectListener( new timeListener() );

levels.addSelectListener( new levelListener() );

Arguably having the choice objects as parameters to the GridController constructor is unnecessary coupling of the classes, so I then decided to eliminate those parameters, and do the wiring in the visual builder, in this case Visual Cafe. Unfortunately, I could no longer use inner classes, because the methods in the inner classes

public void itemSelected( SelectEvent e);

are not found by bean introspection, nor apparently can it be added in a BeanInfo object. I need three SelectListeners in GridController, and only one can be implemented per class. To get around this I would have to add new classes (essentially make the inner classes separate from GridController). Inner classes are preferred not only because are contained in a single file, but because they have access to private methods, which otherwise have to be declared package accessible. (I did not implement this solution, so I’m not sure what other problems would arise). There are likely other designs that deal with the problem of needing multiple listeners of the same event type but different event source.

It may be that the visual tools might be improved to see inner classes (the beanbox also has the same limitation), or there may be some fundamental reason why they cannot. To some extent it underscores the limitations of the "visual builder" approach to specifying interactions: it simply can’t match the power and safety of coding the interactions directly. (From my limited exposure to IBM’s Visual Age, they have a more powerful visual paradigm for specifying the parameters to the calls triggered by events. However I did not have time to test if they had a way around this problem, and in any case they don’t currently support inner classes at all.)

More fundamental is the question of how best to specify complex interactions between components. Visual tools generate adapter classes in response to point-and-click user actions. Anything other than simple interactions will likely require a deeper understanding of the component API’s. The ability of visual builders to provide enough context to the user to make this easier than coding is unproved.

In the same vein, the original implementation of GridController contains all of the interactions between components in one place, and allows the use of inner classes to elegantly partition the work. In contrast the adapter classes generated by the visual builder are less coherent and possibly harder to maintain. Further prototyping on more complex applications are needed to understand whether wiring components together in a visual builder is a viable way of building applications.

While the wiring of components in a visual tool has some questions, the layout and composition of the components in a visual tool was straightforward. In my mind this satisfies goal 3 (Section 4) of enabling assembly in a visual GUI builder.

If component software development simply means paying attention to the way software is packaged and assembled to maximize code reuse, then it will undoubtedly be a success. Whether the scope of its success will be limited by other technological developments is unforeseeable. In any case, one or more of the major component worlds, COM, JavaBeans, and CORBA will be the enabling environments for component software.

It will be some years before the real benefits of component software development can be readily assessed. As in any engineering discipline, execution is at least as important as concept. The failure of any specific project or even domain may be due to a number of complex factors, interactions, and intangibles. The success of component software development for building scientific analysis and visualization applications will depend on three factors:

First, there must be an active pool of users and developers with the technical knowledge and the motivation to create component software. Szyperski notes that component software development is harder, not easier than traditional software development:

While freely shared components may not be required to have as stringent requirements as commercial ones, components do require at least one more level of abstraction in their design. To undertake this additional effort, application developers will need to be convinced of the overall usefulness of the concept, likely by prototypes and demonstration projects.

Second, a component application framework for scientific analysis and visualization must be created that allows components to be developed independently and to interoperate. Such a framework not only specifies the way that components interact, but also provides supporting libraries that make component creation easy by providing commonly needed services and helper classes.

Third, semantic and syntactic descriptions of the data objects to be shared by the components in the framework must be specified. These objects must closely match the data objects and concepts that application programmers are already familiar with. New semantics and levels of abstraction must be carefully documented and supported by helper classes in the framework.

To continue this work, a semantic description of meteorological data, along with a component framework to support component development, must evolve simultaneously. The best way to do this is a rapid iterative development alternating between data interface specification and implementation. The very simple prototype implemented for this project indicates very little of the shape that either the description or the framework must have. In the Appendices is the first real version of the "Grid" data object, and the next step is to implement it using components.

There are other important issues to investigate in developing jmet, including resource discovery and remote access of distributed data archives. The Java Activation framework might be worth using as a model for data decoding. These are not component issues per se, however.

An important question to answer (and the subject of an ongoing debate within Unidata) is whether to assume that we will implement jmet on top of the VisAD class hierarchy, or whether we will define abstract data types in the form of interfaces that will hide the implementing class. VisAD is a very strong package of Java classes on which to build scientific visualization applications, and there’s no question that there will be an implementation of jmet using VisAD. However, there are good theoretical reasons to try to avoid any VisAD dependency, so that the implementer is free to choose any implementation. Continued prototyping is needed to make clear whether the interfaces can hide the implementation without sacrificing the power and flexibility of VisAD.

Finally, while the use of a GUI application builder for component composition and layout is a clear win, there are still questions remaining about when the specification of component interactions should be made in a visual builder, and when it should be programmed directly. This too requires further prototyping to answer.

(Alper96) N. Alper and C. Stein. Geospatial metadata querying and visualization on the www using Java applets. In Proceedings of Visualization '96, pages 77-84. IEEE, Oct. 1996.

(Buschmann96) Frank Buschmann, Regine Meunier, Hans Rohnert, Peter Somerlad, Michael Stal, Pattern-Oriented Software Architecture : A System of Patterns John Wiley & Sons, 1996

(Butler89) Butler and Pendley, A visualization model based on the mathematics of fiber bundles, Computers in Physics, Sep. 1989

(Davis94) Davis, G. P., and R. K. Rew, 1994: The Unidata LDM: programs and protocols for flexible processing of data products. Preprints: 10th International Conf. on Interactive Information and Processing systems for Meteorology, Oceanography, and Hydrology. Nashville, Amer. Meteor. Soc., 131-136.

(Hibbard94) W. L. Hibbard, B. E. Paul, D. A. Santek, C. R. Dyer, A. L. Battaiola, and M-F. Voidrot-Martinez, Interactive Visualization of Earth and Space Science Computations , Computer 27, No. 7, July 1994, 65-72. W. L. Hibbard, C. R. Dyer, and B. E. Paul, A Lattice Model for Data Display , Proc. IEEE Visualization '94, 1994, 310-317. W. Hibbard, C. R. Dyer, and B. Paul, Display of Scientific Data Structures for Algorithm Visualization, Proc. IEEE Visualization '92, 1992, 139-146.

(Infospheres96) A World-Wide Distributed System Using Java and the Internet. IEEE International Symposium on High Performance Distributed Computing, Syracuse, New York, August 1996.

(LSA98), R. Bramley, D. Gannon, T. Stuckey, J. Villacis, J. Balasubramanian, E. Akman, F. Breg, S. Diwan, and M. Govindaraju. Submitted for IEEE book on PSEs, March 1998.

(McGrath97) Robert E. McGrath, The NCSA HDF Java(TM) Interface: Design and Implementation., NCSA 26 March 1997. http://hdf.ncsa.uiuc.edu/hdf/java/hdf/design.html

(Sluice97) Isenhour, Shaffer, Begole, Nielsen, and Abrams, Sluice: A Java-Based Framework for Collaborative Interactive Modular Visualization Environments (Virginia Tech) online paper at http://simon.cs.vt.edu/linc/public/Sluice/

(Snyder 86) "Encapsulation and Inheritance in Object-Oriented Programming Languages" by Alan Snyder, OOPSLA'86 Conference Proceedings, pages 38-45, September 1986

(Szyperski98) Clemens Szyperski, Component Software, Addison-Wesley Longman, 1998

(VisAD97) Hibbard, Ruedan, Rink, Emmerson, Fulker, and Anderson, The VisAD Java Class Library for Scientific Data and Visualization, draft, Sep 8, 1997.

(Wulf89) Wulf, W.A., 1989: The National Collaboratory‹A White Paper. Appendix A in Toward a National Collaboratory, an unpublished report of a National Science Foundation invitational workshop held at Rockefeller University. and 1993: The collaboratory opportunity. Science, 261, 854-855.

(Yorick) ftp://icf.llnl.gov/pub/Yorick/yorick-ad.html

Appendix A. Mathematical Basis for Coordinate Systems in Scientific Datasets

Appendix B. Semantics of Meteorological Data