| • |
|
|
|
|||
|
||||
6.4
USING MCIDAS-OS2 TO INGEST AND DISPLAY DATA SETS AVAILABLE ON UNIDATA'S INTERNET DATA DISTRIBUTION (IDD) SYSTEM
Don Murray*
Unidata Program Center, Boulder, Colorado
*Corresponding author address: Don Murray, Unidata/UCAR, PO Box 3000, Boulder, CO 80307;e-mail <dmurray@unidata.ucar.edu>
1. BACKGROUND
For the past decade, Unidata has been helping colleges and universities acquire real-time meteorological data. One effort has been to provide a specialized (Unidata-Wisconsin (U-W)) data stream of satellite imagery, surface and upper air observations and gridded data sets in McIDAS compatible format, generated at the University of Wisconsin's Space Science Engineering Center (SSEC). It was initially distributed to Unidata sites by satellite transmission as a 9600 baud circuit and ingested using PC-McIDAS. The slow speed of the satellite-broadcast circuit put limits on the number and size of products that could be included in the stream. Another problem with this data stream has been that the conventional data (surface, upper air) is limited to decoded observations for North America and the gridded data is limited to NGM output and global analysis grids. For many Unidata sites using McIDAS, this has narrowed the view of the capabilities of the system.
In 1994, Unidata started distributing the U-W data stream on its Internet Data Distribution (IDD) system (Baltuch 1997) which effectively removed the bandwidth constraints on the data stream and provided quicker transmission of the products. Other data streams including the National Weather Service Family of Services (FOS) circuits, National Lightning Detection Network (NLDN) flash summaries, and NIDS products were also distributed on the IDD. These data streams provided Unidata sites access to a number of new data sets.
A large percentage of IDD sites uses UNIX workstations running Unidata's Local Data Manager (LDM) software (Davis and Rew 1994) to ingest, process and/or relay these data streams on the IDD. Unidata provides the LDM-McIDAS software for converting some of these data sets into McIDAS compatible formats, which can then be accessed using McIDAS-X or McIDAS-OS2. For small colleges and departments, the overhead of running a UNIX workstation (cost and administration) can be daunting. Many of these sites have opted to run the PC version of McIDAS (McIDAS-OS2) to provide a low cost/maintenance system for meteorological analysis and display. The version of McIDAS-OS2 distributed by SSEC has several modes of ingesting and accessing remote data sets, but does not include software for ingesting data from the IDD streams. Unidata developed a program (McLDM) which enabled those sites running McIDAS-OS2 to act as leaf (receive-only) nodes on the IDD and receive the U-W stream. Initially, there were no programs to process the data from the other streams.
2. XCD for OS/2
SSEC developed the McIDAS-X Conventional data Decoder (XCD) software (Pyeatt et al. 1995) to receive and process data from the FOS data streams on McIDAS-X workstations. This package consists of a set of server routines that include ingestors to read text or binary data from serial ports and write it to staging files and data monitors which run one or more decoders and process the data in the staging files into McIDAS compatible files (MD, GRID). The raw and decoded data sets can be accessed through a set of client commands that run on both McIDAS-X and McIDAS-OS2 workstations. Unidata distributes a modified version of McIDAS-XCD to allow data ingestion directly from the LDM.
McIDAS-OS2 sites not running the LDM software needed a way to access the data sets on the other IDD streams. The source code for the client routines was common between McIDAS-X and McIDAS-OS/2 and since the server routines use McIDAS library or XCD client library functions for many of their processing calls. Unidata decided to explore the possibility of running the XCD servers under McIDAS-OS2. Only minor changes were required to get the XCD server program working under McIDAS-OS2 and ingesting FOS data from the PC serial ports. The programs were ported and tested using a Pentium-class PC.
The next step was to create a way of ingesting the circuits from the IDD. McLDM was modified to request and receive multiple data streams and to write the data to named pipes. The routine used to read data from the serial ports was modified to be able to read from named pipes as well. With these modifications, Unidata sites who were only using McIDAS-OS2 could receive and process data from the FOS data streams in addition to the U-W stream.
The only FOS data stream that presented a major problem was the High Resolution Data Stream (HDS) which contains numerical model output in GRIB format. Since GRIB is a bit-oriented format, the McIDAS-XCD GRIB decoder could not be used because of the byte-ordering differences between the supported UNIX platforms and OS/2. A rewrite of the GRIB decoder was necessary.
Since the IDD has data streams other than those supported by the SSEC McIDAS-XCD package, Unidata needed to develop programs for processing the products on these other streams. Decoders were developed for the NLDN and WSI NIDS data streams and interfaced with the other McIDAS-XCD programs. Since both of these streams consist of binary data, the existing XCD binary ingestor routine was used to read these data from the named pipes and store them in staging files. Two new data monitors were written to process the data in the staging files. The decoder portions of these monitors were based on existing decoders for these data products distributed with the LDM-McIDAS package.
Additional modifications were made to the decoding routines to perform Unidata specific functions (e.g., updating system tables). Some of the McIDAS-XCD client routines were also modified for increased ease of use and some new client routines were added.
3. ACCESS TO NEW DATA SETS
McIDAS-OS2 has always had the ability to ingest products from the U-W data stream. For many Unidata McIDAS users, the view of the data available for analysis and display was limited to the products in this stream. With the addition of McIDAS-XCD for OS/2, the view for small colleges and universities running only McIDAS-OS2 expanded to include global observations and new types of data. In addition to the imagery, gridded data and wind profiler data available on the U-W stream, sites can now ingest and decode products from the following circuits:
The XCD for OS/2 server package has decoders for the following types of data:
that can be displayed in McIDAS-OS2 using SSEC and Unidata-developed commands. In addition to the decoded products above, sites can access:
using the XCD client applications in the Unidata McIDAS-OS2 distribution.
As additional data sets and data streams are added to the IDD, routines will be developed for accessing, decoding and displaying them.
4. CONCLUSION
The addition of the McIDAS-XCD for OS/2 software has greatly expanded the data sets that small schools can access without the overhead of running a complicated UNIX system. The software can be run on an inexpensive Pentium-class computer running IBM's OS/2 operating system. The built-in networking of the latest versions of OS/2 allows sites to easily share the data sets among several workstations running the McIDAS-OS2 package. The processing and display capabilities of McIDAS make this a powerful tool for atmospheric education and research.
These OS/2 systems can also be used as WWW servers to provide small college campuses access to real-time weather information. An example of this can be seen on the Unidata McIDAS demonstration machine (http://shecky.unidata.ucar.edu).
5. REFERENCES
Baltuch, M. S., 1997: Unidata's Internet Data Distribution (IDD) System: Two Years of Data Delivery. International Conference on IIPS, Long Beach, 13, Paper 6.6
Davis, G. P., and R. K. Rew, 1994: The Unidata LDM: Programs and Products for Flexible Processing of Data Products. Preprints, International Conference on IIPS, Nashville, TN, 10, 131-136
Pyeatt, J, and P. Wahner, R. Kohrs, J. Hagens, T. Wittaker, and J. Rueden, 1995: Observational Data Ingest Under McIDAS-X. Preprints, International Conference on IIPS, Dallas, TX, 11, 376-377
| Contact Us • Site Map • Search • Terms and Conditions • Privacy Policy • Participation Policy | ||||||
|
||||||
