Policy Committee Meeting Summary:
September 13-14, 1999

Arlington, Virginia

Participants

Members	Representatives
John Merrill (Chair) Ken Crawford Robert Fox David Knight James Moore Charles Murphy Mohan Ramamurthy Julie Winkler UPC Staff Sally Bates Ben Domenico Joanne Graham Jo Hansen Linda Miller Russ Rew	David Carlson (UCAR/ATD) Harry Edmon (ATAC) Jack Fellows (UCAR/UOP) David Fulker (UPC) Mary Glackin (NOAA) Bernard Grant (NSF/ATM) Clifford Jacobs (NSF/ATM) Jennie Moody (Users Com.) Tim Spangler (UCAR/COMET) George Serafino (NASA) Others Susan Jesuroga (UCAR/COMET)

Administrative Matters

• The Committee welcomed Mohan Ramamurthy as a new member and thanked Jim Moore and Ken Crawford for their contributions. New member Mike Biggerstaff was unable to attend (due to field program involvement) and expects to participate in the next meeting. Charlie Murphy is the new Policy Committee liaison to the Users Committee.
• Correction to last meeting's summary: Under International Data Issues, the summary should reflect that Knight's response to MetEirann's objections was to block access to his data by any except .edu sites.
• The next meetings of the Policy Committee will be:
  • January 27-28, 2000 in Boulder
  • May 22-23, 2000, possibly at SSEC in Madison, Wisconsin
    

Status Reports

Director's Report
Copies of Fulker's status report and Graham's budget report were distributed at the meeting. Fulker's assessment was that there were few problems in most areas. Caution is warranted in three areas: User Relations (because some users have expressed a preference for greater emphasis on legacy applications and on AWIPS), Data Distribution (because the rapid growth in data distributed via the IDD is straining the system), and in Finances (because projections indicate a depletion of reserves in the future, exacerbated if any advancement projects are attempted).

Discussion

• There was a great deal of discussion on the scalability of the IDD and the reliability of the LDM. More data are being queued for longer periods and data over an hour old are lost. At the same time, the amount of data being pushed is increasing rapidly. Some of the limitations of the LDM are: queuing is limited to one hour and is limited in size to 2 GB; it supports a very limited number of data feed types; changing an LDM's data source is handled manually (except where individual universities have implemented ad hoc fail-over scripts), and hence the IDD topology is essentially static and it is maintained through human effort (even though automation could yield better results).
• The ability to prioritze data was discussed. The UPC doesn't track data utilization. Filters are already employed at leaf nodes; however, relay nodes may not employ filters because they must relay whole data streams.
• It was suggested that the ATAC might want to discuss IDD scaling. It was also suggested that scaling the IDD represents a fundamental computer-science problem in networking technology. Jacobs suggested that the problems be identified and a presentation on this be made to CISE as groundwork for a proposal to CISE for additional funds to help address the problems.
• Graham noted that she has established a Web page (accessible from the Unidata home page) for organizing next summer's Users workshop

Users Committee Report

Moody noted that the Users Committee had not met since the last Policy Committee meeting. The next meeting will be at the end of September and will focus primarily on next summer's workshop. Moody reported that she's asked the committee via e-mail for comments on interest in and access to satellite data. (A copy of her e-mail is in the notebook.) One respondent suggested replacing McIDAS satellite data with NOAAport GINI imagery; she answered by observing that the GINI images (as a result of remapping, etc.) are unsuitable in many research contexts.

Murphy's status report on MetApps is in the notebook.

Discussion

• Someone suggested carrying both GINI and McIDAS satellite information on the IDD. It was noted that this would add significantly to the IDD burden.
• Fox noted that SSEC is willing to change the contents of its data stream. It could easily add more data without additional costs; the limits are not contractual or technical, but result from Unidata decisions on data-stream contents. He also noted that SSEC's ADDE server allows another approach for accessing satellite data. ADDE, however, is limited to those in the community who use McIDAS.
• Fox reported that GOES-9 and -10 data will be rescued to 3590 tapes by the end of this calendar year, and the rescue of GOES-8 data should be completed by the end of March 2000. The completion of the GOES-8 rescue means that all GVAR data, which spanned the timeframe from 1994 to the current date, will be available on 3590 tapes for cheaper and more rapid access. Work will then commence on rescuing GOES modes A, AA, and AAA data (GOES data prior to GVAR) from 1978 to 1996.
• Concerning MetApps, it was noted that, despite the addition of new members, the number of people actively involved in discussions hadn't changed. The idea of rotating membership was mentioned. Murphy noted that UMADA was working well and had been used successfully over the Spring and Summer 1999, but Knight disagreed, noting that although UMADA delivers all comments as mail to the participants, participants must reply using the Web-based system, which is cumbersome and limits discussion. Overall, however, the MetApps/developer arrangement seems to be fruitful. Rew reported that the developers are happy with the input they're receiviing.

NOAA Report
NOAA Report Glackin report that all AWIPS systems have now been deployed and the agency is in the process of commissioning them. The NWS has continued to close older, smaller offices as planned. Only 22 remain to be closed. In the process of affecting these changes, some interesting developments have occurred. For the Willeston, ND, office the NWS will be adding a commercial S-band doppler radar. For the Erie, PA office, the NWS will use the FAA radar. It will be an ASR-11 to be installed late in 2001. Data from the new commercial S-band doppler will be added to the NOAAport data stream.

NOAA has also released its strategic plan, available on line at: www.nws.noaa.gov/sp/

This plan recognizes the provision of climate services as part of the NWS mandate and emphasizes science and technology infusion in the form of USWRP, community-based modeling efforts, and the C-STAR program.

The latest information concerning NIDS is in the notebook; the agency is testing its central server and working to produce radar mosaics and on adding radar data to NOAAport.

In other areas, NWS is addressing how to handle the projected increases in model data volumes. One approach may be to divide the model data into geographical sectors similar to what is being done on NOAAPORT now with GOES East and GOES West. The agency is also trying to establish standards for decoders.

Discussion

• The schedule for adding radar mosaics to NOAAport depends on the NWS being ready to do so, not on agreements with the NIDS vendors.
• NWS plans to push as well as broadcast their data; they're testing purchased software (not the LDM) to push data; they are increasing the bandwidth allotted for this.
• Unidata's agreement with WSI to distribute NIDS data is renewed annually.

NASA Report

Serafino reported on

• The Status of EOS
  • Landsat has been launched and data are available
  • QuikSCAT has been launched; data will be available early next year.
  • Terra is now scheduled to be launched next month. Instruments aboard include MODIS, MISR, ASTER, MOPITT.
  • 1-degree assimilation test flow of EOS data will begin next month
  • EOS PM-1 is still scheduledfor launch at the end of next year
  • End-to-end testing continues on EOSDIS Core System (ECS) at the DAACs
  • Access to data will be through DAACs, ESIP (Earth Science Information Partners), RESACs (Regional Earth Science Application Centers), remote sensor application data centers, and others; there is no policy of charging for EOS data.
• Data Access Issues
  • Goal: making data holdings interoperable at several levels (users applications, systems applications, data usage, acquisition, and searching)
  • Researching mechanisms for reducing volumes to maximize information content (content-based searching with coincident data searching); goal is to reduce/mine data before distribution or to reduce/mine data within the processing stream (e.g., by obtaining algorithms from users)
• Employing "Beowulf Cluster"--PC-based parallel processsing cluster.
  

NSF Report

Jacobs reported that NSF is engaged in searching for replacements for GEO and ATM directors and that there is no news on the NSF budget. Grant reported that 8 equipment grant proposals were funded (19 proposals had been received). Since the panel only awarded $77,000, the remaining monies will be carried over into the next year. He noted that the review panel found an unusual number of the proposals to be substandard.

Discussion

• The equipment-grant system isn't broken, the community simply needs to help sites develop appropriate proposals. A buddy-system was suggested, as was phone calls to sites.
  

COMET Overview

Spangler and Jesuroga gave an overview of the COMET program.

Discussion

• COMET is moving to AWIPS and doesn't have the resources to support both GEMPAK and AWIPS. They are considering moving case studies to the lowest common denominator format (e.g., GRIB). The next case study, however, will be in both GEMPAK and AWIPS formats
• COMET was commended for releasing the case studies and for developing GARP. Both have been most welcomed by the Unidata community.
• Community contributions may provide a mechanism for continuing the creation of GARP case studies.

GEMPAK and AWIPS

Fulker noted that the letter from Gen. Kelly on GEMPAK (in the notebook) affirms that the NWS is ceasing development of GEMPAK and eliminates any doubt about UPC flexibility to use the GEMPAK code as it wishes. This means that McIDAS and GEMPAK are now equally available to Unidata participants without the need of licensing. The letter does indicate that a few enhancements to GEMPAK will be forthcoming before development ceases entirely. The letter also affirms that the NWS is interested in cooperating with universities. Fulker believes that cooperation may now be in the realm of decoder development.

For some users, the implications of this letter cast doubt on the viability of GEMPAK and GARP in Unidata, considering our prior reliance on NWS and COMET staff for advances. In contrast, Fulker believes that the Unidata staffing in this arena will yield continuing advances (as already demonstrated, e.g., in adaptations of GEMPAK to utilize new data sources) though the pace will slow.

Fulker asked the committee to consider this matter from a policy perspective: Is it appropriate to continue the present priorities, which sacrifice the pace of advancement in legacy software (GEMPAK and McIDAS) so as to achieve more quickly the benefits of Java-based MetApps software? A closely related policy question is whether Unidata should directly exploit other software from NWS and FSL, such as the anticipated Linux version of the AWIPS workstation. Fulker advocates maintaining the current plan and priorities, and he said the benefits of the Java/MetApps strategy include: high levels of platform independence; a component architecture (which allows users to "get under the hood..."); use of abstract data and display models; less emphasis (than in AWIPS, e.g.) on powerful data-base servers; and greater potential for the direct integration of Unidata software with other educational materials that are likely to emerge as universities exploit advancing technology.

Discussion

• Moving GEMPAK into maintenance only is a real concern to a number of users. This is balanced by the fact the the Users Committee reaction to AWIPS was relatively cool
• The focus of universities is to teach the concepts of meteorology. Since most students will never work in the Weather Service, some felt that the loss of shared applications may not be a problem for many campuses.
• AWIPS software is available to universities but is dependent on platforms (HP workstations) that are beyond the means of most universities. It also represents the struggle of universities between being cutting edge or looking like a Forecast Office. Universities and FOs have different missions but both want to continue having at least some commonalities. FSL and NWS are considering a port of AWIPS to Linux, which may make AWIPS more available to universities. However, AWIPS does not fit the Unidata model: it's a turnkey system that is not user configurable.
• The GEMPAK situation exemplifies UPC vulnerability to losing support for packages developed by others: Vietor's support for WXP essentially has disappeared, Mary desJardin first moved from NASA to NOAA and now will focus on AWIPS. SSEC also has suffered a significant reduction in resources for software development. This reflects a more general trend away from software development toward programmatic foci, often accompanied by the use of commercial packages such as IDL. This is true at NCAR as well: ATD has dropped development of Zebra.
• Groups within NCAR use SCD's visualization tools and IDL. IDL is widely used in part because it can be used to analyze netCDF files. Hence Unidata's efforts have actually facilitate the use of IDL, and this underlines the importance of well-designed interfaces, built on effective, useful abstractions. Separating software levels appropriately through such interfaces not only allows more collaborative work, but also ensures that others WILL write applications.
• Without shared applications, there is no Unidata community.
• The committee agreed that retaining access to COMET case studies is a major concern. UPC funding for case study support is level, but the UPC is committed to ensuring accessibility by universities.
• The loss of GEMPAK development may not become a problem for some time. Some noted that universities are slow to maximize their use of applications capabilities; by the time most fully utilize GEMPAK to the point of desiring additional capabilities, MetApps may be operational. Others were concerned that MetApps development may take much longer than this anticipated lag time.
• There was disagreement on whether community contributions would really appear. Experience indicates that universities won't step in to do development, that development needs to be centralized. This was countered by the observation that community members are actively involved in building on VisAD.
• It was suggested that if MetApps development is not proceeding as rapidly as predicted, this needs to be identified early on and additional resources found to aid in the development work.

Participation Policy

Fulker explained that the purpose of the draft document (distributed by e-mail) was to codify who it is the UPC should support. The method currently used to track users (the software licensing method) is out of date.

Discussion

• One purpose of a new participation policy is to provide guidelines to UPC staff on how to respond to queries from people ouside of Region 4. The desire to document participation is also driven by the need to account for resource use to the NSF (to measure success).
• UPC needs to decide what information it needs/wants. For some software (netCDF), there is no way of determining who is "participating" beyond simply tracking downloads.
• The committee warned the UPC that there may not be any way to design a fully automated system. This is particularly true when data are involved. Does downloading software allow you to subscribe to data?
• Virtual universities pose a risk of being unable to identify who should be supported--there may be no single representative. Fulker noted that Unidata currently encourages and supports one feed per university, so funneling support should not be a problem
• If you create a database of users, then you must be prepared to open the database to individuals under the Freedom of Information Act.
• Access software, access to support, and access to data may be separate issues. Does participation include those who access all UPC services? In terms of resources, the crux of defining participation is who is able to request support. Automation of the LDM would result in the topology of the IDD being self-determined to a large extent.
• Unidata applications software is not in the public domain. GARP is copyrighted by COMET; McIDAS is copyrighted by SSEC and the UPC is allowed to distribute McIDAS to users for educational and research uses only. These restrictions must continue to flow through to users. COMET uses "point-and-click" licenses that advise users of the existence of the copyright and the restrictions on use.

Action 1:

The UPC will develop a prototype user tracking information system and report the status of this effort at the next meeting.

Action 2:

Fulker will refine the draft Participation Policy and this will be an agenda topic at the next meeting. The refinement needs to strengthen the section on points of contact in respect to data flows and include more on Unidata's core constituency.

Action 3:

Merrill will write an article for the next issue of the newsletter on the increase in data usage, the potential future difficulties these pose for the LDM, and staff efforts to meet these challenges. The article will include information on possible solutions.

Strategic Implications

Fulker noted that the level of growth in the IDD is leading the UPC to questions its current priorities regarding LDM development. The UPC had assumed that data flows would be limited by Internet bandwidth, but this has become dfficult to identify (e.g., the Abilene project's interest in CONDUIT). Furthermore, the SSEC no longer links the size of the McIDAS stream to a specific costs.

Fulker noted some possible approaches:

• (less desirable) Assign data priorities, with guidance from Users Committee.
• (more desirable) Develop mechanisms such that local decisions can be made in the context of local capacity. This probably requires creating MANY data feeds.
• Nearly any solution will require LDM development, including automated routing, larger queues, a simpler LDM-administration interface (perhaps in conjunction with reimplementation in Java), and the ability to handle more feeds.

Discussion

• It is not clear whether the transfer of NIDS to NOAAport will create a problem for the LDM. The importance of success is illustrated at Oklahoma, where NIDS are the most requested data by at least one order of magnitude. One indicator suggests that no problem will arise: NCAR/RAP currently handles NIDS from all radars with a single LDM.
• Policies on data use are easier to set. Restricted data are distributed separately. But the rapid increase in the amount of available data results in two problems: bandwidth limitations and mechanisms for easy selection by users of portions of the stream.
• Millersville is already acting as an intelligent filter for some satellite data. In addition, the UPC might consider developing a simple application to allow users to select radar/satellite/model/sounder/... by geograpical area.
• It was very clear in the context of all the discussions that the Policy Committee felt that the current priorities vis a vis support for legacy applications and MetApps development is appropriate.

Action 4:

LDM development will be a topic on the agenda for the next meeting.

Action 5:

The UPC will inform the community of the status of GARP and GEMPAK.

Action 6:

Data use policies will be a topic on the agenda for the next meeting.

Action 7:

Fulker will draft a policy on the distribution of NOAAport data by Unidata..

Action 8:

In consultation with SSEC, Unidata will draft a software distribution approach, based on shrink-wrap licensing, for Policy Committee consideration in January.

Action 9:

An overview of the SuomiNet proposal will be on the agenda for the next meeting.

---

Unidata Homepage

---

This page was Webified by Sally Bates.
Questions or comments can be sent to <sally@unidata.ucar.edu>.

This page was updated on .