Community Newsletter
Winter 1995

Table of Contents

 


Soon, GOES Imagery via Internet Only!

by Dave Fulker, Unidata Program Director

This is a reminder about the planned cessation, at year's end, of the broadcast via satellite of GOES images and other data prepared for Unidata at the University of Wisconsin-Madison. At that time (1 January 1996) the only way for Unidata universities to receive the Unidata/Wisconsin data stream (also known as the McIDAS channel) will be to participate in the Internet Data Distribution (IDD) system.

I write this article having just used the World Wide Web to look at the current performance of the IDD system. The last report shows that 55 universities received 100% of the products they requested during the preceding 1-hour period (14Z on 22 March), and for most of them this represented about 11 MB of data. (NIDS data add another 6 MB, but there are still relatively few subscribers.) Therefore the IDD, in the aggregate, transferred well over 500 MB of information perfectly in the single hour I examined.

A handfull of universities show lesser percentages and not all hours are as good as the one I examined, so there's still much room for improvement, but the effectiveness of the concept has been verified, and our commitment is undiminished to adopt IDD as the method of choice for data transmission.

Terminating the satellite broadcast of the Unidata/Wisconsin data stream by the end of the year is appropriate for three reasons: (1) our experience leads us to believe universities can make the transition to IDD and rely on it by the end of the year, (2) resources spent on the broadcast could be put to better use, such as offering greater coverage from the two satellites (GOES-7 and 8) now in orbit, and (3) the broadcast channel used to convey the Wisconsin service is running near capacity, so the addition of needed data, such as East Coast coverage from GOES-8, is impractical without a significant change. In contrast, our experience indicates that the IDD, as presently configured, can disseminate additional data.

However, we need community cooperation and commitment to succeed in making this important change. If you anticipate problems in using the IDD, or if you need more information to do so, please send e-mail to support@unidata.ucar.edu or point your Web browser at the Unidata homepage and look at the IDD-related items there. In any case, please allow adequate time to be sure your IDD connection is running properly well before the satellite-broadcast sunset; in my view, this means you should initiate data reception via IDD no later than the end of this summer. Thanks for your cooperation.


Plymouth State College's Campus Weather Display Systems

Editor's note: This paper is condensed, with the permission of the author, from a paper of the same title.The original version contains information on where to get the applications used and links to the scripts Jim uses to create his displays.

Background

Plymouth State College (PSC) was one of the first 20 schools set up to implement the satellite broadcast of Unidata PC-McIDAS data. From the beginning meteorology faculty have been extremely interested in providing widespread on-campus distribution. We envisioned this distribution of weather information not only as good public relations and advertising for our relatively new bachelor of science degree program, but also as a means of enhancing the meteorological education of majors, non-majors, and the overall campus community.

The Initial System: A first step!

In February 1992, Robert Loring, then one of our undergaduate students, installed a new version of WXP on the meteorology department's IBM RS/6000 Model 320; he also installed Unidata's Local Data Manager (LDM) to handle data ingest (we had previously used the WXP ingest program).

We wanted to have campus-wide access from any terminal or a PC or Mac connected to the campus mainframe computers (even via modem). Since graphics on vt100-type terminals were unsatisfactory, we stuck to text products. We also wanted some basic plain language and/or easily understandable products of interest to the community. We finally decided on the following text products:

  • Forecasts:
    • Local forecasts prepared by senior meteorology majors
    • State forecast from the NWS office in Concord
    • US city forecasts from the ABUS 11-14 summaries
  • Current Observations:
    • For each individual New England state
    • For the entire US
    • Ski reports for the Northeast (FWUS01 KALB)

Loring set up the LDM headers file upon ingest to route the state forecast from Concord, the ABUS summaries, and the ski reports to appropriate filenames (e.g., con_summary, us_summary, and ski_report for user access. The appropriate files are always overwritten upon receipt of new data. Initially, UNIX commands were used to display the files to users; pg was used for longer files and cat for shorter ones.

For the observations, Loring modified the WXP sfclist program to list the full city name instead of the 3-letter identifier for the New England observations. A second modified version lists both the city and state information instead of the identifiers when listing information for the entire US. The US list is usually piped through the UNIX pg command for display.

An initial script starts upon login to our WEATHER display, and another script sets local time variables and starts the WEATHER menu program. We originally used a UNIX menu program developed locally by Ted Wisniewski of our Computer Services to generate professional-looking, full-screen menus, but are now using Lynx as the text interface. I also set the UNIX cron scheduler to automatically convert raw .sao files at 4, 8, 12, and 22 minutes after each hour to assure that the converted files are timely.

PSC's WEATHER system went on-line in early April 1992. During the first year, the system was averaging 20-25 remote connections per day during the regular semesters. There appeared to be a good mix of student and faculty/staff use. Since then, activity has been increasing steadily.

The Second System: Graphics at Last!

Nearly every major academic building on our campus contains a room of PCs (mostly IBM PS2/30s), which we viewed as potential graphics platforms. These PCs are on the campus network (DECNET) and boot from a file server. The server then runs menu software to provide students with easy access to a number of applications on the file server.

Surprisingly, the text file generated on the RS/6000, a UNIX platform, initially caused the largest headaches because it contains line feeds and carriage returns that are incompatible with the DOS machines. Ted Wisniewski came to my rescue again and put together a short program that takes the UNIX ASCII file and puts in the appropriate control characters for DOS. For the WEATHER menu system, I stayed with the menu program that was already being used by the school.

With help from Unidata staff, I was able to identify the parts of the Unidata prototype campus weather display scripts that allowed me to build almost any WXP graphic into a GIF file. Later I found other ways to build the same images, such as using the WXP 4.7 programs that allow me to generate GIF files as output. Now I could automatically build GIF images via WXP on the RS/6000.

In generating and testing GIF files, I found using ppm routines created the most reliable script for generating satellite images. Initially output from the ppm routines was sent to me as countless e-mail messages; some Unidata folks offered a redirection solution that took care of this problem. For multiple overlay maps, I use scripts patterned after one Dan Vietor sent to the community.

Next, I found a shareware program, called FastGIF, that allows me to display GIF files on DOS machines. As advertised, this viewer is fast. It works well with all except satellite images (it doesn't seem to handle greyscales very well) and it can easily be incorporated in an interactive DOS script. I build all WXP-generated GIFs at a resolution of 640x480 (to maintain a sharp image) with no more than 16 colors (since most of the display machines have only minimal VGA capability). After the GIF files are produced, they await automatic FTP, which is scheduled on another machine.

The last major hurdle involved the display of the greyscale GIF satellite images that FastGIF does not handle well. I ordered another set of shareware programs called Imagepro that contained two programs that were particularly useful. The "show" program displays the satellite image and preserves greyscale contrast. When creating the satellite image GIFs, I set the map outline color to white in order to stay within the "show" program's 64-grey limit for 640x480 images. The "key" program holds the image on the display until any key on the keyboard is pressed. I also purchased an upgrade of this package, called Multimedia, that had additional special effects for displaying and erasing images.

The PSC Campus Weather Center went online in mid-November 1992. It provided widespread, interactive access to updated graphical weather products via PC clusters around campus. (Since one of these clusters is located within our science building, it also provides us an opportunity to modernize some of our general-education, weather-laboratory-course exercises.) This service is very popular at PSC and accessing it is the first example on using the cluster facilities in an orientation course required of all first-year students.

Several clusters on campus have since been upgraded and configured to run the Microsoft Windows Mosaic interface. There are also several Mac LCs on campus that are configured with Mosaic and have similar capabilities.

Lobby Display: Look Ma, no hands!

One of our most successful ventures at distributing weather information on the PSC campus is the automatic display located in the lobby of our science building; it is appropriately named LOBBY. The purpose of this display is not only to advertise our program, but also to educate viewers on some of the display-and-analysis capabilities available, while showing them the current weather conditions. This site was chosen for its visibility: the entrance to the admissions office is less than 25 meters away and the lobby is the waiting area for PSC's largest lecture hall.

LOBBY uses a subset of the GIF products already generated for the PSC Campus Weather Center, along with information screens in PCX format created with the MS Windows Paint program to the appropriate size (VGA: 640x480). The program cycles through approximately seven minutes of weather graphics and explanatory text.

For our display device, we use one of our original PS2/60s with an SMC EtherCard Plus (attached to the campus network), running OS/2 Version 1.3 and PC/TCP Network Software for OS/2. The display actually runs in the DOS window using a continuously looping script with embedded Multimedia 1 program calls.

After the GIF files are generated on one of the RS/6000s, they are automatically FTP'd to the PS2/60 to refresh the data for display. The PS2/60 is slow, so to keep the display program from trying to show a file while the file is being updated, I instituted a renaming protocol, which doesn't take place until after an image has been displayed and then only if a new image is available.

We started LOBBY in early June 1993, and it has run nonstop except for a few power outages and hardware failures. Our Computer Service staff has expressed an interest in locating other displays throughout campus using some of their old PS2/30s--it is an excellent way to use an older machine.

The WWW Mosaic Server

In April of 1994, we changed our system by setting up a World Wide Web home page for the text-based products. It is quite easy to use and configure, once the Mosaic server software has been installed. I had the first version of our Web pages up in about four days.

The Web has several benefits: there is a single menu for the same range of data and the Web browsers such as Mosaic/Lynx do not require an account for access and use far fewer server resources. The Web technology also moved us to automate the creation of the files for current converted surface observations (in the earlier system the commands were executed via a user's menu selection). This has greatly speeded up the response time for users.

The GIF and text products that I was already building served as the basis for my initial server. Within weeks, I was adding a little more glitz and more products, including mpeg movies using mpeg_encode and mpeg_play software. In just days, I was able to make loops and add them to the server. (I've found that the accessory program xplaygizmo provides greater control for viewing these loops than mpeg_play). Be aware that encoding mpegs does require lots of computer resources.

You can find our server here.

Addendum (from a personal communication)

This past fall, Margaret Hamilton, a student, worked to expand our Web server as part of her senior project. Our main objective is to provide additional tutorial information for middle- and highschool teachers who may know little about meteorology, but who want to be able to bring it into their curricula or better understand the products they are accessing. Our first offering in this area was the PSC Meteorology Program Cloud Boutique, which provides information and pictures of common cloud formations.

A second objective is to add more information specifically about our own meteorology program. Hamilton used material from the college catalog, program brochures, etc. It was very easy.

Most of the pictures on the server were taken with an Apple QuickTake camera, which stores the images digitally in memory. They can be downloaded to either Macs or Windows PCs using specific versions of the Apple QuickTake software. You can shoot 32 low-resolution images (320x240) or 8 high-resolution images (640x480) or a combination of each before downloading. The camera hooks up to a serial port and the software is used to download the images. Their native format is .qtk; you cannot save directly to GIF files, but the software allows you to save the images as JPEGs, bmps, and some other formats. I use several packages to postprocess the images (e.g., sharpening, brightness control, etc.).

PSC is currently working with various schools throughout the state to provide Internet access and the Mosaic software installation. We see most schools obtaining this or similar access over the next several years and we want to be ready to serve their needs.

Happy computing!


Using the Web in the Classroom: A Texas A&M Experience

Last fall I initiated a course called Weather Forecasting that will be a requirement for first-year meteorology majors and transfer students. It's a one-unit seminar-type course that will now be offered every fall. It meets in our Laboratory for the Exploration of Atmospheric Processes (LEAP). This laboratory was funded through the Unidata equipment-grants program and the NSF Instrumentation and Laboratory Improvement program. It contains fifteen SGI workstations, including a high-speed workstation with a video board for projecting images and making videotapes.

The course is taught almost entirely with Mosaic. It has its own homepage containing links under three main headings:

  • "Local Links" consists of our WWW-based fully interactive weather interface (for accessing DD plus information), our suite of GEMPAK analysis and forecast maps, and a pointer to the department homepage.

  • "Data Links" is a collection of pointers to some of the most frequently used Web sites, such as Purdue (WXP), Illinois, and Florida State (satellite ground station).

  • "Today's Imagery" contains images created just for that day's class.

For the "Today's Imagery" section, I will log in a couple of hours before class and check out the current weather situation. I'll identify an interesting topic, appropriate for students with only a high-school education, download a few appropriate images and maps, and establish some local links in this section. I'll also load in a few forecast products pertinent to the day's forecast. For example, last fall there was a binary typhoon (two tropical storms spinning around each other) in the western Pacific. We followed the system for days, using imagery obtained via the Web from servers in Australia and Japan.

Class begins with a fifteen minute lecture/seminar, using both a whiteboard and the images placed on the homepage. Next, a student leads a forecast discussion focused on the current city in the National Collegiate Weather Forecasting Contest. Finally, the students examine data on their own and make their own forecasts that they enter using a form on one of the Web pages.

Most of the imagery we access is generated elsewhere. This is one of the big advantages of the Web and the Internet--you no longer have to produce what you want locally; you can let other people do the work and let your students take advantage of it. Through the Web, they have access to raw model output, National Weather Service forecast discussions, forecast maps, current data, and satellite imagery.

This course would not have existed without the Web. For our incoming class, it has replaced a full year without meteorology with a semester of forecasting and weather discussions on topics of immediate interest. I hope it has also replaced boredom with enthusiasm and given students the means to satisfy their curiousity about the atmosphere.

Late afternoons during the fall, anyone on the Web can access the course homepage and find what I regarded as the day's most enlightening imagery. Here is a link to the class homepage (website no longer available).


Data Descriptions Available on the Web

by Linda Miller, External Programs Coordinator

If you have been following the development of the Unidata WWW pages you may have noticed a bulleted item titled "Data" on the Unidata homepage. The Unidata Users Committee encouraged us to provide descriptions of the data available to Unidata sites. This information has been gathered from many sources and covers many data topics. It includes a description of the NWS Family of Services that Unidata sites acquire from Alden. NEXRAD Information and Dissemination System description and data stream details also contain a reference to WSI's home page. In fact, we try to reference other home pages and information that is native to the source to provide the most current information available. Lots of information has been provided on the Unidata/Wisconsin data stream and on the National Lightning and Detection Network. The information will continue to evolve, so keep checking.

You can find the page here.

Here's some of the data information you can access:

  • Data Volumes
  • Numerical Modeling
  • FOS Product Catalog for DDS and PPS: Data Contents and Keyword-searchable Index
  • WMO Header: Key to Abbreviations
  • FAA 604 product codes
  • National Weather Service Forecast Zone Information
  • METAR Reports and TAF Forecasts: Information and Update
  • NWS Station Modernization Update
  • New NWS Automated Surface Observing System (ASOS) Site Commissions
  • Automated Weather Observing Systems (AWOS) Information
  • National Weather Service Telecommunications Gateway (NWSTG) Gopher Server

Package News Is on the Web

Unidata's application packages are in a constant state of flux; bug fixes are made and new features are added continually. Traditionally each of these changes and additions has been accompanied by an e-mail announcement to the appropriate list; this meant that lots of people received messages about bugs they'd never seen or features they might not ever use.

We are beginning to move away from this system and towards using our Web server to keep users apprised of our progress. Each package is listed on the Unidata home page and under each package is a link to a "What's New" page for that package. On these pages you will find running accounts of the fixes and improvements that have been made to the packages.

You should make it a habit to periodically check the "What's New" pages for the packages you use. The latest information will always be available there.


Earthquakes on the Net

Editor's note: This article is reprinted, with permission of the author, from the electronic newsletter TidBITS, #261, 30-Jan-95.

by Adam C. Engst

Around 7 PM last Saturday night, just as our furnace kicked on, the house started to roll. We have a relatively old house, definitely too old to learn new tricks like rolling over and playing dead. After three or four seconds, we realized that the furnace hadn't blown up and a second or two afterwards that, in fact, we were having an earthquake. We scurried under a doorway, but I promptly left on a four-foot rescue mission to save Fred, a 20-year-old cactus that I was not going to leave to the tender mercies of the quake. Luckily, the house decided to stop after 10 or 15 seconds. It was over. No damage, no breakage, no loss of power, gas, water, or TCP/IP access (although we promptly went and checked everything).

Then, rather than turning on the TV or the radio to see what had happened (heck, we knew what had happened--we wanted details), we went to the Mac and out over the Web. Not being a major earthquake buff, I had to go through the excellent Yahoo subject catalog to find the earthquake pages. From there I went to the USGS National Earthquake Information Center in Colorado, which had a Finger to Gopher gateway [Now a Web server. -Ed.] for the latest information on earthquakes.

That was all fine and nice, but since I did this literally minutes after the quake, there wasn't any data about our earthquake. However, in reading the text of the Finger report, I saw a more local machine at the University of Washington. So I ran Peter Lewis's Finger program and fingered quake@geophys.washington.edu. The first time it only had automated information that it claimed couldn't be trusted, but that information remained constant after the warning went away.

Now we knew that the earthquake had been a magnitude of 5.0, and that it was a bit southwest of Seattle. But where exactly? Then I remembered the Xerox PARC Map Viewer. It took a little figuring out, but I managed to get the proper search phrases to locate and mark the exact epicenter of the earthquake.

Perhaps the most interesting part of the quake experience for us was using the Internet to find this information and immediately send it to friends and family. Suddenly, after about the fourth message we sent out, a piece of mail came in from someone in Australia, who happened to hear about the earthquake from someone near Seattle reporting a problem with Anarchie. Obviously, had this been a serious earthquake, the Internet connection would have gone down, at which point no communications would have been possible. Still, the Internet made the world feel much smaller at the time, and somehow, that was comforting.

And speaking of much worse earthquakes, a large amount of information about the Kobe quake appeared on the Web rather quickly, and there's a nice collection of it at Yahoo.


Virtual Community Comes Together fro UCAR's 35th Anniversary

by Ben Domenico, Unidata Program Center

UCAR is holding an open house to celebrate its 35th anniversary. Events will take place on April 21 (dress rehearsal in the afternoon) and April 22 (most of the day Saturday). The open house coincides with Earth Day and National Science and Technology Week.

As part of the open house, Unidata would like to enable visitors to "take a ride on the information highway" and get a sense of what's going on at Unidata universities. One way we will do this is to have workstations running Mosaic and Netscape that will allow visitors, with the help of the staff, to follow pointers to the homepages and weather servers community members have constructed.

The theme of the open house is "35 Years of Discovery"; please let us know if you have something special on your WWW server that fits the occasion. Send an email to Sally Bates with any URLs you think visitors would find interesting.

We're also planning to have live video-conferencing using the CU-SeeMe program from Cornell. We have heard that some universities have used it for teaching classes from remote sites, so we are planning to hold a few weather briefings from sites that are set up with CU-SeeMe. If you are experienced with this technology and would like to participate (or have some advice or ideas on how to go about it effectively), please contact me:

bdomenico@unidata.ucar.edu
(303) 497-8631


Arrivals and Departures

by Matt Hicks, Unidata Program Center

There have been several changes in the makeup of the Unidata staff since the last edition of the newsletter lumbered into the mails.

Those of you using the WXP software may already be aware of the departure of Mike Wright, whose tireless efforts with not only WXP, but with the Unidata information servers, the K-12 outreach efforts, and IEIS support and development went well beyond what anyone could reasonably have asked. Mike has become the third employee of a successful Boulder startup company involved in optical instrumentation and AI systems. We wish him well in this new endeavor.

Our office mom and do-it-all administrative assistant Linda Henderson has also moved on, bumping herself upstairs, literally. Linda, who kept things running around the UPC for almost three years, has taken a job with NCAR's HIRDLS program which occupies the offices above the UPC. Linda will be missed not only for her organizational skills, but also for her winning personality, her eagerness to help people no matter what their problems, and her frequent contributions of cakes and cookies with which we here at the UPC had an ongoing love/hate relationship.

Many of you will get to know newcomer Cindy McCombs in the next few months as you prepare to travel to Unidata meetings and workshops. With Linda's departure, Cindy became the front office staff after only a month in her new digs. Cindy comes to us after a stint with the Resolution Trust Corporation, which has apparently become irresolute and will close its Denver operation this month. Cindy is a Colorado native who enjoys running and teaching aerobics; we'll be trying to cajole her into leading classes during breaks at the next workshop.


Troubleshooting the IDD

As of this writing, we have approximately 70 Unidata sites on the Internet Data Distribution (IDD) system and we expect about another 50 sites to join by the end of the year when the satellite broadcast of the Unidata/Wisconsin channel ends. The quick growth of the IDD has led to an explosion of support contacts regarding problems with data reception. Our experience has shown that most of these questions involve problem isolation on the IDD network itself. While it is relatively easy to tell if your LDM server is up and running, it is more difficult to find network and relay node problems.

There are two basic symptoms that manifest themselves when there is trouble on the IDD system. You are either not receiving any data or you are suffering unacceptably large losses. Both of these could indicate problems with the network; data losses could also indicate a problem at your site.

Help! I'm not receiving any data!

Assuming that you have satisfied yourself that your LDM server is running properly, if you lose your data feed completely, you should first check the activity at your upstream feed site. The easiest way to do this is to use the notifyme utility, which will tell you if you can reach your upstream site and whether that site is itself receiving data.

If notifyme fails to get a response from your feed machine, you can use the ping utility to determine if the upstream machine is "alive." If you're not receiving data, but ping indicates the upstream machine is alive, then the LDM on that machine is probably not running. The only thing you can do then is to contact the site administrator at the upstream node with the problem. You should also copy the UPC on the e-mail message, so that we are aware of the problem.

If ping fails to reach the upstream machine, run the trace-route utility. The traceroute utility indicates the route a network packet takes to reach the specified host. If there is a network outage, the traceroute will hang at some point and the last host displayed in its output will give you some idea of where the outage is. You should then contact your local network administrators, give them the information you have gathered and let them take up the problem with the regional network provider.

If notifyme gets a response from the upstream machine, but indicates that that machine is not receiving data, direct ping and ldm-ping to various nodes feeding the upstream machines (you can determine which machines are upstream from you by checking the routing information available on the Unidata Web server). Then contact Unidata user support with the information you have gathered and we can take it from there. Providing the information obtained from ping and ldmping may help us correct the problem faster.

Help! I don't receive all of the data!

The problem of poor data reception is a much more difficult one to identify and solve. Experience has shown us that two main problems are usually responsible for chronic data loss: a computer that has a heavy processing load or a bottleneck someplace in the routing of packets through the Internet.

You can use the top utility (available by anonymous FTP) to examine your system load. This utility displays the top processes--based on cpu usage--running on your machine. It will also show you the average load on your computer. If top reveals that you have a heavily loaded machine, your LDM will probably lose data during periods of high data flow.

If you determine that your machine is not excessively loaded, the next thing to look at is the routing the data takes through the network. The first thing to do is to run ping periodically during a period of several hours to determine if you are losing IP packets. You need to run ping more than once because there are always intermittant outages and bottlenecks on the Internet and you have to make sure that you have a chronic problem, rather than a transient one. If losses are greater than 10% on average, there is probably a network problem.

If you determine that you are indeed losing packets, you need to have someone at the upstream node run traceroute to your machine several times. For each node that the packet goes through, traceroute shows timing information. From this information you can determine if one node in the routing is causing a bottleneck, which usually appears as a large increase in the latency to that node. If you discover such a bottleneck, you need to contact your local network provider for assistance. You should also contact the UPC; in cases where a bottleneck cannot be resolved quickly, we may be able to switch you to a different upstream site so as to avoid the problem. We can also work from our end to help identify and correct bottlenecks.

The reason traceroute needs to be run from the upstream node is that often the routing between two hosts can be radically different depending on the direction the packet is flowing. As an example, a recent series of traceroute tests from the University of Washington, in Seattle, to UCLA showed that packets took a basically direct route down the coast from Seattle to Los Angeles, but packets going in the other direction went from Los Angeles to Hartford, Connecticut to Seattle.

You might also check the Alden Electronics homepage (website no longer available) .

Alden maintains information on the status of their connection to the network, outages at the NWS, and other information you may find of interest and help in diagnosing IDD network problems.

Note that, no matter what the problem or its cause, it takes information to identify and solve it. In many instances just knowing what is in the LDM logs at either the upstream or downstream nodes can help pinpoint a problem for you. But this can in itself be a problem because, for most sites, these logs are not available without contacting the site administrators, and that takes time. To make it easier to get the logs (thus making them much more useful) the UPC encourages all IDD sites to install a WWW server and make their logs available via this mechanism. There are a few IDD sites that already do this and it reduces problem-determination and resolution times dramatically. If you have questions on how to set up a server at your site, please contact Unidata user support and we will be glad to work with you.

One other point about network problems. It is really critical that you get to know your campus networking people. Assuming that you decide that the problem lies with the underlying network infrastructure, they will become your main point of contact in resolving the problems.

Future Work in IDD Monitoring

The UPC is actively working on methods and tools to make the problem of monitoring the IDD easier for everyone. Currently, tools such as ldmprods and idd_monitor begin to address the situation, but there is still much to do. We are beginning to look at utilities that will monitor the state of the IDD and the underlying network and hardware infrastructure on a continuous basis so that long-term trends can be noted and analyzed. We will be releasing these tools as they are completed and we strongly urge all IDD sites to use them.

Other sites are also developing scripts for monitoring and are making them available to other IDD sites. If you have developed your own methods for monitoring, please share them with others. The success of the IDD is based on principles of community involvement and support, and your help in this area is both needed and greatly appreciated.


The IDD Statistics: Seeing the Forest and the Trees

by Robb Kambic with Matt Hicks, Systems Programmer and Documentation Specialist at the Unidata Progam Center

One of the challenges the UPC has had to meet in implementing the Internet Data Distribution (IDD) system has been devising a way to monitor the performance of the system. To this end we created the ldmstatistics package to report on the reliability of both individual LDMs and the system as a whole. It has been said that you can make statistics say anything, but it has been our goal to get them to tell the truth.

If you point your Web browser at the IDD homepage, you will find links to numerous charts, graphs, and tables that provide data on how well the IDD is functioning. Deciphering the information from these sources can be difficult, so we will single out some that you might look at and explain what you will see.

This link will take you to the mother of all tables (link no longer available). This 2.2 MB text file displays the overall reliability of each LDM in the IDD system averaged over the total amount of time for which we have received statistics from that LDM. Due to its size, this file is hard to interpret and almost as difficult to download; it does, however, provide a general idea of how much success each site has had in implementing the IDD system.

Easier to read are various graphical displays that characterize recent and long-term IDD performance. We have developed charts that reflect two fundamental perspectives on reliability. One perspective is systemwide--it is based on the percentage of products actually delivered, out of the total number of data products that could potentially be delivered (summed over all sites requesting those products). A second perspective focuses the performance as seen by the individual sites, categorizing them according to their data reception quality and creating histograms of the sites falling into each category.

The Unidata Web server includes both long-term and 24-hour charts for each of these perspectives. However, we have built two long-term histograms reflecting the day-to-day performance of the IDD. In one, the bars represent each day's performance by describing the typical hourly figure, as represented by a mean; in this case, the number of sites falling in a given category for each day is simply the 24-hour average of the one-hour figures averaged over 24 hours. The other method is to categorize sites according to their 24-hour performance figures; this clearly is the more stringent measure, because no site will fall in the 100% category, for example, unless its data reception was perfect for every hour of the day.

Long-term IDD performance charts may be found on our Web server under the heading IDD Daily Site/System Charts and Reports. The chart titled "Histogram Based on System Performance" was created using the first method, i.e., it indicates typical hourly performance. The chart designated "Histogram Based on Sites Performance" employs the second, more stringent measure. Other graphics on the server are variations of these basic types. (Links no longer available. Current IDD performance can be accessed throughthe IDD homepage.)

Determining the appropriate metric by which to gauge IDD performance is no simple task. We are constantly working to hone and refine the statistics to produce the truest measurement of IDD reliability possible. As such, some fluctuations in apparent system reliability (but by no means all fluctuations) represent much more a change in the measurement of the system performance than an actual change in the delivery of data. Changes in the way the statistics are calculated are recorded in the "What's New" section of the IDD Web pages.

You may notice that IDD reliability actually was lower in February than at earlier times. We are still trying to understand and correct this, though our initial impression is that several factors have contributed. One of the most significant is that the volume of data at the source has risen greatly, due in part to the addition of DIFAX, but primarily because of changes in the content of the High-Resolution Service (HRS), which includes Rapid Update Cycle (RUC) data as well as many other kinds of gridded products. Our charting of data volumes began only recently, so we have no easy means to illustrate the growth graphically, but it has been dramatic, more than doubling the data available to Unidata universities. As can be seen on the Web server under the heading IDD Daily Feed Charts, the system now regularly handles more than 100,000 products per day (measured over all data sources), yielding total volumes that often approach 400 megabytes per day. Summed over all recipients, these numbers suggest that the IDD, viewed as a nationwide, distributed system, will soon be managing more than 20 gigabytes of data per day, as a cooperative university endeavor.

We want you to know that we are watching closely to see how the IDD performs. We have set stringent measures for ourselves and much work remains, but we think the statistics monitoring is an important aspect of achieving our goal: economical and accurate delivery of needed real-time data from multiple sources. We hope you too will use the statistics to watch our progress.


Majordomo Ends General E-mail Havoc

by Robb Kambic, Systems Programmer at the Unidata Progam Center

The UPC is now using the Majordomo software package to maintain its many mailing lists. This package allows users to subscribe to and unsubscribe from a mailing list by using simple commands contained within a mail message. While the use of Majordomo relieves UPC support staff of some rather tedious tasks, it also benefits users by allowing them access to more information about the lists. Information that can be obtained includes:

  • Which lists you are subscribed to.
  • Who is on a particular list.
  • A short description of the list (not yet completed for all lists).
  • A listing of mailing lists at UPC.
  • Help on using Majordomo.

To get a listing of Majordomo commands, send a message to: majordomo@unidata.ucar.edu. Leave the subject line blank and put the following commands in the body of your message: help, end. In the near future we hope to have a Web page that allows you to execute Majordomo commands from within your Web browser.

The List of Lists

In the process of coverting to Majordomo over 40 mailing lists that were considered to be inactive were deleted. The mcidas list was changed to mcidas-os2 because of user and list name conflicts here at UPC. The nps, hds, and hrs list were merged into the datastream list as a general repository for FOS-type problems.

Majordomo is being used with both UPC internal lists and lists that are open to the public. Both types of lists are reported when you request a list of lists from Majordomo. Only the following lists are open to the public:

alpha mcidas-os2
community mcidas-x
datastream ncdigest
dataws needdata
difax netcdfgroup
floater-info nsfdemo
floater-nids nws-changes
fsldata skymath
gembud summary_nsf
gempak-team sun_products
ieis weatherbud
k12 wxp
ldm-users ynot-users

Majordomo commands involving lists not in the above list will be intercepted and approved or disapproved on an individual basis.


GEMPAK and McIDAS Display More Products

Recently released patches and updates allow users of GEMPAK and McIDAS to display new types of data. These additions allow users to make greater use of the data available via the Internet Data Distribution (IDD) system.

A new version of LDM-McIDAS contains a decoder to convert IDD-transmitted WSI NIDS data to McIDAS AREA files. This is made possible by the new nids2area decoder, which is based on work originally done by Harry Edmon of the University of Washington. The AREA files produced by this decoder are the first to include the new "auxiliary block" (AUX) feature added to the AREA specification by the SSEC. The auxiliary block contains ancillary information about the data products.

Both McIDAS-X and McIDAS-OS2 were modified to allow the display of NIDS data in the new AREA file format. Other recent changes to the McIDAS packages provided support for GOES-8 images and the ability to display satellite images in Molleweide projection. Support for NOWrad data should be available soon.

GEMPAK 5.2 users can FTP a patch that allows them to display NIDS and GOES-8 products and overlay them with other GEMPAK graphics. The patch provides GEMPAK support for the new auxillary blocks in the McIDAS AREA format. In order to use the information stored in the AUX blocks, your site must be running the LDM-McIDAS decoders; binary distributions of the package are available via FTP for those not running or not licensed for McIDAS.

In other GEMPAK news, since mid-February the HRS data circuit has been carrying new high-resolution NGM and AVN model output grids centered over the US. GEMPAK users can decode these grids using gribtogem, and display and analyze them using any GEMPAK gridded data programs.

Dan Vietor has also announced WXP support for GOES-8 images. WXP was the first Unidata-supported package to provide display capability for NIDS data.

Be sure to watch the "What's New" pages for the packages you use; patches, bug fixes, and new releases will be announced there as they happen.


NSF Accepting Proposals for Equipment

The Division of Atmospheric Sciences, National Science Foundation, is now accepting equipment proposals from academic institutions engaged in teaching and research in the atmospheric and related sciences for Unidata- supported systems. (System definition assistance is available by contacting the Unidata Program Center at support@unidata.ucar.edu.) Small schools and schools that have not previously submitted proposals are encouraged to apply. It is NSF's intent to assist those institutions that presently do not have interactive meteorological computer capability to participate in the Unidata Program. In addition, NSF will accept proposals for purchasing hardware and software to assist institutions with a transition from OS/2-based to UNIX-based systems and from systems that depend on the KU-band broadcast of the Unidata/Wisconsin data stream to systems that will be capable of receiving those data via the Internet Data Distribution (IDD) system. NSF will also accept proposals for purchasing hardware and software that will enhance an institution's participation in the IDD system. The target date for submitting proposals to be considered for funding in FY 1996 is June 1, 1995.

The following criteria will be used to evaluate the proposals:

  • Contribution to Research. Potential that the proposed equipment will enhance and contribute to local research programs in progress or being developed; potential for the equipment to support innovative and significant research.
  • Contribution to Education. Potential that the proposed equipment will enhance and contribute to local education efforts in the atmospheric and related sciences by providing new approaches to classroom and individual instruction.
  • System Management Competence. Technical soundness of the proposal with respect to equipment selection and integration with existing local systems; capability of faculty and staff involved to manage and utilize the proposed equipment; adequacy of the institutional commitment to assist in obtaining, managing, and maintaining the proposed equipment.
  • Contribution to Unidata Community Capability. Potential for the equipment and associated developments in concepts and software to contribute to the enhancement of the community capability for interactive analysis and computation through Unidata. Commitment to participation in Unidata's community-based support efforts, including but not limited to enhancing or augmenting the IDD system. This includes sites already acting as relay nodes which may need equipment upgrades as well as sites which are well-positioned on the network and have the willingness, expertise, and staff to act as relays but lack adequate equipment.

Unidata equipment awards will provide funds for hardware and the requisite software only. Significant cost sharing for the equipment by the institution is required. Site preparation and maintenance costs should be provided by the institution as well.

Submission of Unidata Proposals

Proposals should be clearly identified for consideration under Unidata. Ten copies should be sent to the Division of Atmospheric Sciences, National Science Foundation, 4201 Wilson Blvd., Room 775, Arlington, VA 22230. Proposals should follow the guidelines specified in Grants for Research and Education in Science and Engineering, NSF 94-2. In addition, proposals should describe:

  1. the relationship of the proposed system to the existing computing facilities, both departmental and institutional;
  2. the percentage of the departmental computing resources that the proposed system comprises; and
  3. the relationship of the proposed equipment to the departmental five-year strategic plan for computing capabilities.

For further information, contact:

Clifford Jacobs
Division of Atmospheric Sciences
National Science Foundation
(703) 306-1521
FAX: (703) 306-0377


NIDS Floater Available

by Russ DeSouza, Millersville University

Late last year, the Unidata program, funded by the National Science Foundation, made arrangements to have a commercial vendor, WSI, distribute NIDS and NOWrad products through the Internet Data Delivery (IDD) system. This agreement allows those users who subscribe for one of the three tiers of service to also contract for three NIDS floater sites for an additional $50. The Unidata Program Office, using the recommendation of the Unidata User Committee, agreed that one site should serve as the administrator of this selection with the hope that some overlap between the satellite floater and the NIDS sites could be realized. Millersville University, the current satellite selection site, will also now be the administrator for this project.

To select the floaters, subscriber sites submit their request to: floater-nids@unidata.ucar.edu This input will be tallied by Millersville and the selected sites will be entered on the appropriate WSI Web page. The Web-page procedure has both advantages and disadvantages: in the plus column, the selected sites can easily be changed during the day if weather conditions change at the initially selected sites; on the other hand, someone has to enter the selected sites even on weekends, rather than requesting them ahead of time as is done with the satellite image).

A rationale for choices will be broadcast each morning (weekdays) to the floater-nids e-mail list.

How to Subscribe

To subscribe to the distribution list, users should send a message to: majordomo@unidata.ucar.edu. Leave the subject line blank, and in the body of the message put only the following lines:subscribe floater-nids, end. Please note that users must already be receiving some sort of product from WSI to be eligible for the floater service. Users are also assumed to have display software. Presently, we are using Dan Vietor's (Purdue) WXP software, in particular radnew, which uses the raw distributed files. McIDAS and GEMPAK can display the products, but they must first convert the raw products to AREA files. This format provides greater flexibility, but at the cost of 7-10 times more disk space.

Unidata Supported Platforms

by Matt Hicks, Unidata Program Center

This table shows the computers and operating systems currently supported for Unidata-distributed software. The specifications for UNIX platforms represent a fairly minimal, LDM-compatible system for data-ingest and file-server duties that also allows color-graphics display for user applications. The OS/2 configurations are only for sites running McIDAS-OS2. Any site budgeting for new equipment should bear in mind that there will be continuing expenses for the data-broadcast services and for hardware and software maintenance. As a rule of thumb, maintenance costs roughly ten percent of the purchase price per year per item. Much more elaborate systems can be purchased to support specific site requirements. You should consult the Unidata Program Center staff before any equipment purchase or upgrade.

* Some Unidata packages (netCDF and LDM4, for example) are actually supported on a wider range of platforms. See the specific package documentation for information on whether the software has been ported and tested on additional platforms.

Please send comments to info@unidata.ucar.edu