AWIPS Forecasting Software

Unidata AWIPS

AWIPSAWIPS (Advanced Weather Interactive Processing System) is a meteorological display and analysis package originally developed by the National Weather Service and Raytheon, modified and repackaged by Unidata to support non-operational use in research and education by UCAR member institutions.

AWIPS takes a unified approach to data ingest, and most data types follow a path through the system starting with an LDM client requesting data from the Unidata IDD. These data files are then decoded and stored as HDF5 and Postgres metadata by EDEX.

Unidata supports two visualization frameworks for rendering AWIPS data: CAVE (a desktop Java application), and the Python Data Access Framework (python-awips).



Install CAVE Viz Client (18.1.1)

CentOS/RH 6 or 7 (x86_64)awips_install.sh --cave Run from the command line
chmod 755 awips_install.sh
sudo ./awips_install.sh --cave
macOSawips-cave-18.1.1-1.dmg
awips-python.pkg
install both
Windowsawips-cave-18.1.1-1.msi
Full CAVE Installation Instructions...

Install EDEX Data Server (18.1.1)

CentOS/RH 6 or 7 (x86_64)awips_install.sh --edex Run from the command line
chmod 755 awips_install.sh
sudo ./awips_install.sh --edex
Full EDEX Installation Instructions...

2018 AWIPS Training Workshop

October 18 - 19, 2018 (Thursday and Friday)

See Training Workshop Schedule for All Unidata Packages

This two-day workshop will familiarize users with the CAVE (client) and EDEX (server) components of AWIPS, tracing data ingest from beginning (raw files) to end (data visualization), giving users full control of a virtual AWIPS system to learn not only how to use the AWIPS tools, but understand how data comes into the system and is processed, and how to investigate and solve data-ingest issues in a real-time environment. Technologies touched on will include LDM, IDD, Apache HTTP, the Qpid message broker, PostgreSQL/psql, Linux shell (bash) commands, HDF5, and Python.

Day 1: CAVE Perspectives & Data Plugins

  • Install & Configure CAVE
  • D2D Perspective
  • Maps, Views, Projections
  • Bundles and Procedures
  • Import/Export
  • Localization Perspective
  • NCP Perspective
  • NSHARP
  • WarnGen
  • GOES-16 (GOES-R) products
  • Importing GIS/Shapefiles
  • Volume Browser
  • Radar Tools
  • Editing User Menus
  • Changing Localization Sites

Day 2: EDEX Server Administration & Data Ingest

  • Install & Configure EDEX
  • EDEX Start and Stop
  • LDM Feed
  • Distributed EDEX Environments
  • Scaling EDEX for High Volume Data Ingest
  • Data Distribution Files
  • Data Purging and Retention
  • Ingesting New Grids
  • Monitor EDEX Users
  • Map Overlays
  • Python-AWIPS API Examples
  • Qpid Message Broker
  • PostgreSQL/PSQL Database Administration
  • Case Study Archiver/EDEX Archive Servers

AWIPS Workshop Requirments

You are encouraged to bring your own laptop computer for this workshop. AWIPS CAVE installs and runs on Windows, Mac, and Linux (Red Hat) systems, so you should download and install the latest client release prior to beginning.

This workshop will use Python programming, Docker image containers, git version control software, and the bash terminal.

64-bit CentOS/Red Hat 7 Linux

Mac

Windows

AWIPS Data in the Cloud

Unidata and XSEDE Jetstream have partnered to offer a EDEX data server in the cloud, open to the Unidata university community. Select the server in the Connectivity Preferences dialog, or enter edex-cloud.unidata.ucar.edu (without http:// before, or :9581/services after).

EDEX in the cloud

Note: Unidata does not support the operational NWS version of AWIPS. It is recommended that only Unidata AWIPS clients connect to Unidata AWIPS servers. Using a NWS client with a Unidata server will result in errors.

Distributed Computing

AWIPS makes use of service-oriented architecture to request, process, and serve real-time meteorological data. While originally developed for use on internal NWS forecast office networks, where operational installations of AWIPS can consist of a dozen servers or more, because the AWIPS source code was hard-coded with the NWS network configuration, the early Unidata releases were stripped of operation-specific configurations and plugins, and released specifically for standalone installation. This made sense given that a single EDEX instance with a Solid State Drive could handle most of the entire NOAAport data volume. However, with GOES-R(16) coming online, and more gridded forecast models being created at finer temporal and spatial resolutions, there was now a need to distribute EDEX data decoding in order to handle this firehose of data.


Software Components

EDEX

The main server for AWIPS. Qpid sends alerts to EDEX when data stored by the LDM is ready for processing. These Qpid messages include file header information which allows EDEX to determine the appropriate data decoder to use. The default ingest server (simply named ingest) handles all data ingest other than grib messages, which are processed by a separate ingestGrib server. After decoding, EDEX writes metadata to the database via Postgres and saves the processed data in HDF5 via PyPIES. A third EDEX server, request, feeds requested data to CAVE clients. EDEX ingest and request servers are started and stopped with the commands edex start and edex stop, which runs the system script /etc/rc.d/init.d/edex_camel

CAVE

Common AWIPS Visualization Environment. The data rendering and visualization tool for AWIPS. CAVE contains of a number of different data display configurations called perspectives. Perspectives used in operational forecasting environments include D2D (Display Two-Dimensional), GFE (Graphical Forecast Editor), and NCP (National Centers Perspective). CAVE is started with the command /awips2/cave/cave.sh or cave.sh

LDM

https://www.unidata.ucar.edu/software/ldm/

The LDM (Local Data Manager), developed and supported by Unidata, is a suite of client and server programs designed for data distribution, and is the fundamental component comprising the Unidata Internet Data Distribution (IDD) system. In AWIPS, the LDM provides data feeds for grids, surface observations, upper-air profiles, satellite and radar imagery and various other meteorological datasets. The LDM writes data directly to file and alerts EDEX via Qpid when a file is available for processing. The LDM is started and stopped with the commands edex start and edex stop, which runs the commands service edex_ldm start and service edex_ldm stop

edexBridge

edexBridge, invoked in the LDM configuration file /awips2/ldm/etc/ldmd.conf, is used by the LDM to post "data available" messaged to Qpid, which alerts the EDEX Ingest server that a file is ready for processing.

Qpid

http://qpid.apache.org

Apache Qpid, the Queue Processor Interface Daemon, is the messaging system used by AWIPS to facilitate communication between services. When the LDM receives a data file to be processed, it employs edexBridge to send EDEX ingest servers a message via Qpid. When EDEX has finished decoding the file, it sends CAVE a message via Qpid that data are available for display or further processing. Qpid is started and stopped by edex start and edex stop, and is controlled by the system script /etc/rc.d/init.d/qpidd

PostgreSQL

http://www.postgresql.org

PostgreSQL, known simply as Postgres, is a relational database management system (DBMS) which handles the storage and retrieval of metadata, database tables and some decoded data. The storage and reading of EDEX metadata is handled by the Postgres DBMS. Users may query the metadata tables by using the termainal-based front-end for Postgres called psql. Postgres is started and stopped by edex start and edex stop, and is controlled by the system script /etc/rc.d/init.d/edex_postgres

HDF5

http://www.hdfgroup.org/HDF5/

Hierarchical Data Format (v.5) is the primary data storage format used by AWIPS for processed grids, satellite and radar imagery and other products. Similar to netCDF, developed and supported by Unidata, HDF5 supports multiple types of data within a single file. For example, a single HDF5 file of radar data may contain multiple volume scans of base reflectivity and base velocity as well as derived products such as composite reflectivity. The file may also contain data from multiple radars. HDF5 is stored in /awips2/edex/data/hdf5/

PyPIES (httpd-pypies)

PyPIES, Python Process Isolated Enhanced Storage, was created for AWIPS to isolate the management of HDF5 Processed Data Storage from the EDEX processes. PyPIES manages access, i.e., reads and writes, of data in the HDF5 files. In a sense, PyPIES provides functionality similar to a DBMS (i.e PostgreSQL for metadata); all data being written to an HDF5 file is sent to PyPIES, and requests for data stored in HDF5 are processed by PyPIES.

PyPIES is implemented in two parts: 1. The PyPIES manager is a Python application that runs as part of an Apache HTTP server, and handles requests to store and retrieve data. 2. The PyPIES logger is a Python process that coordinates logging. PyPIES is started and stopped by edex start and edex stop, and is controlled by the system script /etc/rc.d/init.d/httpd-pypies

AWIPS News & Announcements

More AWIPS news

AWIPS Frequently Asked Questions

Q: What is AWIPS?

A: The National Weather Service is in the process of developing the next generation of its AWIPS software (AWIPS II) to provide a comprehensive package in support of its forecasting and public service operations. This new software will be developed in Java, allowing it to run on more platforms than the current AWIPS software. Many of the underlying technologies in AWIPS will be based on open source projects and the plan is to make AWIPS software also open source. Currently, the NWS National Centers and NWS field forecast offices use different tools to support their mission, with the National Centers using NAWIPS, and NWS forecast offices use AWIPS, which is fundamentally different and not compatible with NAWIPS. The new AWIPS architecture will allow the NWS to reduce development time, expand data access and provide better integration and collaboration between the NWS field offices, river forecast centers and National Centers.

Q: Who is developing AWIPS?

A: Raytheon is responsible for developing the underlying infrastructure for AWIPS and migrating the existing AWIPS functionality into that infrastructure. NCEP is responsible for migrating the existing GEMPAK/NAWIPS functionality into the AWIPS framework.

Q: How is GEMPAK related to AWIPS?

A: GEMPAK functionality, specifically the underlying analysis and diagnostic routines, will be included with AWIPS, most likely as wrapped code. AWIPS is being developed for a "black box" GEMPAK transition, meaning batch scripts and automated processes which call GEMPAK routines would not require updating after migrating to AWIPS.

Q: What is Unidata's role in all this?

A: The Unidata Program Center distributes and provides support for GEMPAK to our member sites. Membership in Unidata is open to U.S. colleges and universities, and is free of charge. GEMPAK is also available as unsupported software without charge to organizations which do not participate in Unidata through the limitations specified in the Unidata participation policy. The UPC release of GEMPAK/NAWIPS incorporates several additions developed both locally and at other locations to enhance the use of real-time data aquired through the Unidata IDD, and through instructional case studies such as those developed at COMET. It includes GARP (GEMPAK Analysis and Rendering Program) which was developed by COMET.

Q: Why is NCEP moving away from GEMPAK?

A: The National Weather Service announced plans in 2007 to cease development of NAWIPS in August 2008 and proceed with a migration of the functionality of that package to the new AWIPS environment. The decision to migrate NAWIPS to AWIPS was made unilaterally by NWS. Since NCEP's primary goal is to support the National Centers, Unidata had no input on this decision.

Q: What is NCEP's plan for NAWIPS/GEMPAK?

A: NCEP intends to migrate all NAWIPS/GEMPAK functionality to AWIPS and intends to make AWIPS available to the Unidata community.

Q: If/When the UPC stops supporting GEMPAK, can I still use it?

A: The GEMPAK source code is freely available and will still be accessible in some form after the UPC ends official support. It will not stop working on any certain date. Furthermore, the existing support materials (tutorial, help manual and documentation) will still be available on line. The gembud mailing list will be kept active so GEMPAK users can provide community support to each other.