GEMPAK Moves Ahead With a New Look

by Steve Chiswell Unidata Program Center

NMAP display with product generation tools.

GEMPAK added a new look last fall with the release of version 5.6 to Unidata sites. This was the first UPC release to feature NMAP, a product generation tool currently used in production by forecasters at the National Centers. NMAP provides many features used in preparing operational products and weather briefings. These same tools can be used by both students and researchers.

Using NMAP's product generation capability, students can analyze and annotate maps on-line. Using the Vector Graphics Format (VGF) for storing the geographic location with each drawing object, products can be created or viewed using any map projection. Now you can import graphical forecast products from NOAA and use or modify them with your own data values. With the graph to grid program, contoured data can be used to generate grid values. Have you ever wanted to change an analysis and have the grid values updated? You can even interpolate the grid values back to station locations.

NMAP now provides eight briefing windows allowing presenters to quickly switch between different products and animation sequences. The "ROAM" feature allows the user to magnify the viewing display while permitting panning of the region of interest. Map and projection selection is made easy through an interactive preview and selection widget. Many other features are available which allow the student to pinpoint a feature, town, or distance. There's even a utility for determining cloud height using infrared satellite images and sounding data.

The power of NMAP derives from many improvements made to GEMPAK since GEMPAK 5.4 was made available to Unidata sites over three years ago. Experienced GEMPAK users will find new device drivers and decoders, additional options and functions, and new programs. Other features, such as file naming templates, will allow novice users to access data with greater ease. Some new functionality, such as the ability to variably depict data and plotting markers according to any data parameter, is readily apparent in commonly used programs such as SFMAP. Still other programs such as GDPLOT2 allow greater flexibility in combining data sets and display characteristics.

Have you looked around the web lately? Many sites are using GEMPAK to produce some really cool products. More Unidata developments such as NEXRAD composites and gridded data trajectories are on the way! For information on the latest developments, see the GEMPAK 5.6 web page at http://www.unidata.ucar.edu/packages/gempak/ GEMPAK5.6/.

Distributing NEXRAD Data in Near Real Time: A View from the Trenches

by Anne Wilson, Unidata Program Center

I’m the kind of person who takes comfort in knowing what, where, why, when, and how things are going to occur. As my uncertainty increases, my comfort level drops. (The relationship between the two is shown in Figure A.) It’s not that the process of acquiring and relaying the NEXRAD feed caused me significant discomfort. It was only a little, really.

To most of our users the transition to receiving the NEXRAD data appeared easy and seamless. Our community has the utmost confidence in our abilities and our commitment. Unquestionably, this expectation of excellent service was met, but, this was the result of much effort on the part of UPC staff, coupled with the timely, responsive help and support of many community members. As with most, if not all, of the tasks we undertake here at the UPC, this one would have been impossible without their assistance.

The Unidata community had long been anticipating the free distribution of the Next Generation Radar (NEXRAD) Level III data. The NWS had established 1 January 2001 as the date it intended to replace the NEXRAD Information Dissemination Service (NIDS) with the Radar Product Central Collection/Distribution Service (RPCCDS), which includes distribution via NOAAport. The UPC expected to distribute the NOAAPORT radar products via the IDD, but the Unidata Users Committee during its November meeting requested that the Program Center, test the distribution to assess the impact of the increased data flow, and so we did.

How much data now?

First, we needed to determine the volume of the data. Some of the data products were available via NOAAport in encrypted and uncompressed format. We needed to know what was on the stream, so we modified the ingest code to place products whose identifier started with "SDUS5" under our new NEXRAD feed type. We then were able to track the current number of products and their size. We found instances of six different products from a subset of the radars. Although not completely representative, this information gave us something to work with and was helpful in our future projections.

Tracking both the number of products and data bytes was a necessity. We compared these figures to the FOS (Family of Services) stream, the minimum data that most sites want. We found that the NEXRAD feed was roughly 50% larger than the FOS feed, and the number of bytes was about 150% greater. We assumed a 2:1 compression rate, understanding that there would be more than twice the number of products on the feed. This is a significant increase in the data volume, especially for small sites that get only the FOS feed.

Our concerns about propagating the data were twofold. First, we were concerned that if a site delivered the data to many downstream sites, it could use up a significant portion of its out-bound bandwidth. Second, we wondered whether sites would get bogged down since we experienced some problems with this within the UPC.

To test the data distribution, we made a path from our server, to Northern Illinois University, to the University of Nebraska, and back to Boulder. We successfully relayed the entire volume of NEXRAD data along this path for a few days, monitoring the load on the machines involved. It was comforting to learn from this test that our top-tier sites could handle the data volume.

How much data in the future?

And then we learned that the data volume would triple (18 new products, not six) by 1 January 2001 and that the worst case (or, best case depending on your perspective), where every radar was in precip mode, would yield around 28,000 products per hour. Yikes?!?

Who Wants What?

This new information made it clear that relaying the entire volume of NEXRAD data to every IDD site was not the right thing to do. Although some sites could handle the load, others could not. While most sites clamor for as much data as possible, the entire volume of radar data contains a lot of "clear air" data which is of limited interest. An alternative was to have sites tell their upstream feeds which radar products they wanted; then only those products would be propagated; but maintaining such a configuration list would be painful for site managers. As weather varied across the country, desired products would vary dynamically resulting in a constant communication and reconfiguration effort. Human behavior is not my specialty, but something told told me this would not work.

Our approach was to make especially interesting radars of the day available to everyone and allow sites to choose a few radars of unique interest.

Towards that end, a process was built that placed products from some selected radars into the data stream under the radar floater feed, FNEXRAD. Leveraging from the existing WSI floater selection process, we enlisted student volunteers at Millersville and DuPage to pick interesting radars of the day for that feed. This is another excellent example of our community’s involvement with this project. We decided to have a topology, separate from the FOS topology, for the rest of the radar products. We would propagate all the radar data around our first-tier sites. Other sites would request products directly from these sites. To keep network traffic manageable, we would ask sites to request products from no more than three radars initially. Once successully tested, we could slowly increase the volume.

The Witching Hour: A Non-Event

By Friday, 22 December we felt we had all the pieces in place that were available to us. We felt we were as ready as we could be. All that remained was to wait and see what happened. I began to feel like this was my own personal millennium bug.

What were you doing just after midnight on 31 December 2000? As a dedicated employee, I was working! Well, for about five minutes. After I woke up my family, who had fallen asleep in front of the TV, (yes, we know how to celebrate a new millennium) and trundled them into their respective beds, I logged on to check out the network.

Guess what was happening? Nothing. Everything was just the same. NEXRAD products were still encrypted and uncompressed. And there was no increase in the number of products.

On the ninth of January at about 15Z, while the number of products stayed in the same range, the volume of data dropped by about 45%. Obviously, the switch had been made. Since the number of products stayed the same, but the size of each product went down due to compression, the propagation of the new feed had no negative impact on the IDD. All in all, it was a bit anticlimactic.

What’s in the Future?

Currently things are in great shape. But, the play’s not over. We understand that sometime in April more products will be added. This will be when the rubber meets the road. Being on Internet2, we expect that our top-level sites will be able to relay the additional 200 to 300Mbytes per hour of data; but, if downstream sites request too many NEXRAD products then latencies over all could increase beyond the acceptable range. For this reason we are asking participants to keep their requests small during this upcoming transition.

It Ain’t Easy

NEXRAD data distribution is a success so far. And, as noted earlier, this is due to hard work and collaboration with our sites coupled with a dash of good luck. In addition to putting the structure in place to propagate the data in the manner we thought best, the slow increase in the amount of data has allowed us to work out kinks. We are now in the best possible position to handle the increased volume in March.

And with that, I'd like to end with a quote from the popular song writer and musician, Dan Hicks: "I’m sure you all think it’s easy standing up here, singing, and playing. But it’s not. It’s not easy. Thank you." Furthermore, without you it would be impossible.

Staffing Changes

We would like to introduce you to three staff members new to Unidata since the last newsletter, to update you on Dave Fulker’s revised status, and to say farewell to a systems administrator.

New to Unidata

Emily Doremire Emily comes to Unidata following a 15-month stint as Site Services Coordinator in UCAR’s Safety and Site Services department. She joined the Unidata Administrative Group at a time when her "can do" attitude was a major asset: specifically as the Program Center staff made its move from temporary offices on the second floor of Foothills Lab IV back into its newly-renovated space on the first floor. Now that the major portions of the move are under control (and we would assert optimistically that is the case) Emily will be the travel “point person” in the program center, and she will assume responsibility for workshop and meeting planning and logistics.
 

When she’s not busy keeping Unidata staff organized and out of trouble, she enjoys the Colorado high country lifestyle in a log house near Nederland. The daily fifty-minute trip down Boulder Canyon gives her the opportunity to catch some extra ZZZs while someone else does the driving.Emily promises not to unleash any killer attack roaches on Unidata staff as long as she is treated well.

Kitty Ferguson What’s this? Tigger the tiger in the UPC? Sea creatures too-a lobster, a whale, a crab, a stingray and that pesky octupus? You will find these critters perched on Kitty Ferguson’s monitor, and her bookshelves. On her desk hutch you can’t help but notice a large RED parrot with silvery tinsel dangling from its wings. You could say (conservatively) that Kitty has quite a menagerie.

Though she’s not new to UCAR, Kitty is new to the Program Center staff since the last newsletter. And, although her present corporate home is the UPC, her time is split between Unidata and the Digital Library for Science, Math, and Engineering Education (DLESE). Prior to assuming the UOP position, Kitty worked in the High Altitude Observatory (HAO), most recently as a Unix and PC Systems Administrator.

Kitty has played, and continues to play, an active role in the wider UCAR corporate world. She presently serves on the Desktop Systems Advisory Committee as an observer, and previously was its vice-chair and Strategic Plan sub-committee member.

When not working, Kitty pursues her interests in Celtic/Scottish history and music, spends time with her family: two kids (a grandchild is on the way) and her mother. She also engages in Fantasy Gaming and Adventure Role-playing. The sea creatures in her menagerie reflect a commitment to scuba: as a participant and as an instructor.

Stop by and introduce yourself to Kitty and her menagerie; but be careful where you sit, there may be a lobster on your chair. Yikes!

Scott Kehoe Scott Kehoe, brand new to the Unidata Program Center and to UCAR as well, is the newest addition to our software engineering crew. He was hired to bring our web site to the technological forefront, and he will be the technical key to making PlazaElectra a reality. We have no doubt that he will adapt in short order to the UPC "culture."

Born in New York, Scott has spent most of his life in Colorado and has called Boulder home for the past year and a half. His most recent work experience is in developing E-commerce sites for a local Boulder company. He interjects that he is happy to be in a scientific and educational setting.

Scott is very serious about his work, but like most Unidata staff, has numerous outside interests. He hikes, plays guitar, and is thinking about pursuing a Class A (beginner) skydiving license this summer.

As fascinating as these facts are, one in particular makes him a natural fit with the rest of our staff. What’s that? It’s his self-professed affinity for the amber beverage with the frothy white top so loved and idealized, really, by all of Unidata’s techies.

Departure

Sandy Whitesel Sandy’s last day at Unidata (and indeed, in Boulder) was 23 March. We’re going to miss him, of course (who could ever equal Sandy’s computer wizardry mixed with a healthy portion of general zaniness), but Sandy is off to Albuquerque, New Mexico. There he will pursue what has become a consuming passion for drum playing. We wish Sandy well.

On Sabbatical

David Fulker Unidata Director, David Fulker, is on sabbatical leave from the Program Center for several months. His present plans are to return to his position at the end of July 2001. The leave allows Dave to focus on developing strategies, exploring opportunities, building partnerships, and writing proposals that serve Unidata’s future interests.

Prior to his departure, Dave delegated the major portion of his normal decision-making activity to Ben Domenico, who is serving as Acting Director. He moved his offices to the third floor in DLESE space, and negotiated a short-term lease on his office space in the UPC.

Committee Column