Introduction and Background
The Unidata Program Center (UPC) makes the National Centers AWIPS package (N-AWIPS, of which GEMPAK is a primary component), currently developed at the National Centers for Environmental Prediction (NCEP), available to the university community. The UPC provides additional features (e.g. GEMPAK Analysis and Rendering Program - GARP, Level II radar support, decoders for special datastreams) in the Unidata release of this software. GEMPAK/N-AWIPS, which is used by nearly 300 universities worldwide, remains one of the most widely used analysis and visualization package that Unidata supports.
The National Weather Service (NWS) is in the process of developing the next generation of its AWIPS software (called AWIPS II) to provide a comprehensive package in support of its forecasting and public service operations. AWIPS II will include migration of AWIPS I application and components into a Service Oriented Architecture, plus will include N-AWIPS (NMAP, NCEP Product Generation software, GEMPAK), data decoders and encoders, Display Two Dimensions (D2D), Graphical Forecast Editor (GFE), Community Hydrologic Prediction System (CHPS), Weather Event Simulator (WES), and a thin-client to support Incident Meteorologists like Fire Weather forecasters. Many of the underlying technologies in AWIPS II will be based on open source software, enabling NWS to make AWIPS II software also open source. AWIPS II also represents a merging of N-AWIPS and AWIPS into one system, enabling hardware and software consolidation as well has enhancing efficiency. Currently, the NWS National Centers and NWS field forecast offices use different tools to support their mission, with the National Centers using N-AWIPS, and NWS forecast offices use AWIPS, which is fundamentally different and not compatible with N-AWIPS. The new AWIPS II architecture will allow the NWS to reduce development time, expand data access and provide better integration and collaboration between the NWS field offices, river forecast centers and National Centers. Raytheon is the primary contractor for developing the AWIPS II architecture and baseline AWIPS I migration, while NCEP holds the primary responsibility for migrating N-AWIPS/GEMPAK functionality into the AWIPS-II architecture, leveraging Raytheon architecture and baseline functionality wherever possible.
NCEP’s N-AWIPS to AWIPS II migration activities are broken into four major areas:
It is worth noting that all N-AWIPS functionality, including GEMPAK, will be preserved in AWIPS II; i.e., all current GEMPAK capabilities and applications will be available in AWIPS II. Moreover, AWIPS II, which will store all data in HDF5 format internally, will no longer require GEMPAK file format for I/O, although GEMPAK will still be supported externally. As part of its N-AWIPS migration, NCEP ceased further development of N-AWIPS/GEMPAK, with the exception of bug and emergency fixes, in August 2008. However, since that time, some development effort has been made to allow GEMPAK programs to read data from the AWIPS II database. A final release of N-AWIPS/GEMPAK has been postponed from October, 2009 to December, 2009.
The university community has a keen interest in the AWIPS II software, especially the migration of GEMPAK to the new system. As such, the UPC staff has been working closely with both NWS Office of Science and Technology, the primary organization managing and overseeing the development and deployment of AWIPS II, as well as NCEP, which is responsible for migrating N-AWIPS components to AWIPS II. The NWS/OST and NCEP view Unidata as an important stakeholder and collaborator in migrating the university community to AWIPS II. As Don Berchoff, Director of NWS Office of Science and Technology, stated in his 12 May 2009 letter to Unidata, “One of our primary objectives as we move forward is to strengthen our partnership with the university community to enable faster and more effective infusion of new science from research to operations in AWIPS. We look forward to our continued partnership with Unidata to meet this objective.”
Considering that Unidata’s primary mission is to enable education and research in universities, Unidata’s principal interest in the AWIPS II effort is in the migration of GEMPAK and related software, including Unidata additions to NCEP’s N-AWIPS release, to the AWIPS II framework.
Status and Schedules from NWS and NCEP Perspectives
The AWIPS II effort is both ambitious in scope and complex in its development, with inter-dependencies across several organizational entities. As such, it should be recognized that schedules for development and deployment will continue to evolve and time-tables will change as the work progresses. Below is a brief timeline of important stages to date and future schedules in AWIPS II migration from the Unidata perspective:
Note: The schedule stated above is according to a message from J. C. Duh, NWS/OST, dated 18 Aug 2009.
Further details on the status of and schedules for AWIPS II development are provided in the following section, along with implications for the Unidata community.
Unidata’s Role and Issues to be Considered:
The migration of N-AWIPS/GEMPAK to AWIPS II and the decision by NCEP to cease further development of the current N-AWIPS/GEMPAK application has, understandably, resulted in heightened anxiety amongst some GEMPAK users. At the same time, many in the Unidata community also have a great deal of interest in the new AWIPS II software that includes the migrated N-AWIPS components. When AWIPS I was developed in the 1990s, it contained some proprietary software that precluded its broader availability to the academic community. The university community clearly does not want to see the same thing happen with AWIPS II system. Unidata is aware of the benefits to the academic community of keeping pace with the highly leveraged AWIPS II software.
The concerns surrounding GEMPAK are two fold: the continued availability of the current GEMPAK application and Unidata’s support for it going forward. To address the concerns as well as the interests of its users, Unidata has been actively involved in discussions and working closely with NWS/OST and NCEP to ensure that the needs of the university community are considered during the migration of N-AWIPS as well as in the development of the AWIPS II software. It is worth repeating that both NWS and NCEP view Unidata as a key partner for testing and evaluating AWIPS II software for education and research use and a focal point for disseminating AWIPS II to the academic community. An important step in establishing a close collaboration between Unidata and NCEP, particularly to help with the migration of N-AWIPS to AWIPS II and to strengthen partnership on the CONDUIT project, was taken earlier this year when Brent Gordon, Chief of NCEP/NCO’s System Integration Branch and the person overseeing the N-AWIPS migration effort, joined the Unidata Users Committee as an NCEP representative to the committee. Unidata’s efforts to make AWIPS II software available under an open source license yielded positive results when the NWS notified Unidata in May 2009 that the U. S. government has removed Raytheon’s copyright markings from the AWIPS II software and that AWIPS II software would eventually be made available to users under an open source licensing agreement.
In addition to ongoing negotiations with NWS and NCEP on the distribution of AWIPS II software, the UPC staff have been participating in the monthly(now weekly) N-AWIPS Migration Teleconferences to track the project’s progress and have also been attending AWIPS II training sessions when appropriate. As stated earlier, the UPC has been receiving pre-release (only binary) versions of the AWIPS II software through a non-disclosure agreement. Presently, the UPC staff is evaluating the AWIPS II software for use by the Unidata community.
Meanwhile, Unidata has been actively engaging its governing committees in developing a migration plan for N-AWIPS/GEMPAK and providing regular updates to its community as developments occur. Both the Policy Committee and the Users Committee have endorsed Unidata’s plans for GEMPAK and AWIPS II and helped develop position papers and announcements to the community describing Unidata’s efforts. In one of those announcements, Unidata reaffirmed its commitment to continue support for the current GEMPAK software for 18 months after the first complete release of N-AWIPS integrated AWIPS II to the National Centers and to the university community. Currently, that release of AWIPS II is planned for the first quarter of 2011, so Unidata’s support of GEMPAK will continue until mid 2012. Although GEMPAK support from the UPC may cease at that time, the source code and documentation will continue to be available to the community so users can continue to use GEMPAK beyond that time period. Also, the UPC-managed gembud support mailing list will be kept active as long as necessary so GEMPAK can eventually become a community-supported package where users provide support to each other.
Despite the aforementioned positive outlook for AWIPS II, including the migration of N-AWIPS/GEMPAK to the new framework, a host of issues remain to be addressed before distributing the software more broadly. Those are outlined below:
Update: Per a message dated 9/7/2009 from John Vogel, NWS/OST: Lightning data are still included, as they should be, with each of the AWIPS II releases. We do, however, have a fix in for Raytheon to make it easier to remove lightning data before we turn it over to customers such as you. That change should be implemented with Slice 6, but will have to be tested like any other change. Guidance provided in JC Duh's message, August 18, 2009 is still valid.”
Closing Comments
The availability of AWIPS II software, including the migration of N-AWIPS to the new architecture, is both an exciting opportunity as well as a significant challenge for both the UPC and the Unidata community. Unidata will need to carefully consider all of the significant issues outlined above (i.e., evolving timelines and associated uncertainties, stability of the software, limited availability of the source code so far, system and hardware requirements, and challenges associated with porting and adapting the AWIPS II software for use by the university community). In addition to the technical and logistical challenges, there are resource implications for the UPC that will need to be carefully weighed-in. Furthermore, the AWIPS II transition also remains a delicate public relations matter for Unidata given the myriad uncertainties. Given the sensitivities all around, it is important that Unidata proceeds methodically and deliberately as it engages its community and begins deploying the software in universities.
Based on the recommendation of the NWS, the UPC believes that it would be prudent to wait until mid 2010 to begin engaging a select group of beta users, followed by a broader release to the community in mid 2011. It is hoped that some community members will be a valuable resource in helping to not only test, evaluate, and possibly port the software to other OS and hardware platforms, but also provide other enhancements to the software. During that period, Unidata will continue to provide regular updates to the community on the AWIPS II transition and Unidata’s plans through a range of communication avenues, including the Unidata web site, e-letters, and emails.
Meanwhile, the UPC is also working with its governing committees and community to prioritize and incorporate many GEMPAK functionalities into the IDV so that some GEMPAK users can begin migrating to the IDV. That is a parallel avenue that Unidata is pursuing based on the recommendation by both the Policy Committee and the Users Committee.
In conclusion, there is cautious optimism on the part of all stakeholders that AWIPS II software will become a valuable tool for university and operational communities, one that will enable new capabilities to advance education and research in meteorology.
References:
NWS & NCEP Presentations on the GEMPAK/NAWIPS to AWIPS II Migration
NAWIPS Migration Status (May 2009) Brent Gordon, Chief NCEP/NCO/SIB
http://www.unidata.ucar.edu/committees/polcom/2009spring/nceppres.pdf
AWIPS Technology Infusion, Jason Tuell http://www.unidata.ucar.edu/committees/polcom/2009spring/tuellpres.ppt
N-AWIPS Evolution, Jason Tuell and Steve Schotz http://www.unidata.ucar.edu/Presentations/AWIPS/AE_Overview_NCEP_v2-1.pdf
Integrating NAWIPS into the New NWS Service Oriented Architecture Migrating NAWIPS to AWIPS II, Scott Jacobs, Software Technical Lead
http://www.unidata.ucar.edu/Presentations/AWIPS/AMS%202008%20Briefing%206A-3%20final.pdf
Unidata's GEMPAK/NAWIPS Migration Information
Home page for Migration Information:
http://www.unidata.ucar.edu/software/gempak/nawipsmigration/
Messages from the UPC to the community:
June 2009
http://www.unidata.ucar.edu/software/gempak/nawipsmigration/nawipsmigration_06_2009.pdf
December 2008
http://www.unidata.ucar.edu/software/gempak/nawipsmigration/GEMPAKmigration_Nov08_2.pdf
May 2008
http://www.unidata.ucar.edu/software/gempak/nawipsmigration/nawipsmigration_05_2008.pdf
Unidata's GEMPAK/NAWIPS Migration FAQ:
http://www.unidata.ucar.edu/software/gempak/nawipsmigration/Gempak_Migration_FAQ.html
Official correspondence from NOAA/NWS
Unidata distribution of AWIPS-II software - JC Duh (18 August, 2009):
http://www.unidata.ucar.edu/software/gempak/nawipsmigration/AwipsIISoftwareReleaseAug09.html
Unidata distribution of AWIPS-II software - JC Duh (12 May, 2009)
http://www.unidata.ucar.edu/software/gempak/nawipsmigration/awipsltr.pdf