Testing 13.2.1 Unified Grib Decoder on CONDUIT GFS

Last month we received the first version of AWIPS II which included the new unified grib decoder (13.1.2). The install procedure for 13.1.2 was more complicated than usual - we needed the full 13.1.1 installation plus a 13.1.2 "update" - so around 8 GB of RPMs to manage. 

If you're unfamiliar with what the unified grib decoder is, here's a quick rundown: before 13.1.2, the D2D perspective (for WFOs) and the National Centers Perspective (for NCEP centers) required separate data decoders and database tables for grib messages. D2D used a decoder called grib, while NCP used a decoder called ncgrib. If you didn't want to bog down your system, you could only run one at a time, meaning: depending on your server configuration, gridded model data would only be visible in one perspective, not both.

This first version of the unified grib decoder had a number of problems, likely from the complicated installation procedure (we do it a little differently here compared to a operational forecast office).

Now with the full AWIPS II 13.2.1 release from the NWS, I can finally test our method for increasing the number of decoder threads on CONDUIT data. This is the same method used to handle ingest of the entire NEXRAD3 feed (~190 radar sites), and is described in detail on the page linked at the bottom of this post.

As delivered, AWIPS II will take roughly 4 hours to decode the high-resolution CONDUIT 0.5 degree GFS with 4 threads, and the message broker bottleneck would get worse with products simultaneously being decoded by other plugins (obs, satellite, radar, etc.)

Unidata's solution to this is to increase the allocated number of threads for the grid (formerly called grib) decoder, a task which involves either re-building the EDEX core RPMs, or editing the already-installed plugin-specific jar archives.    I managed to incorporate the new decoder thread settings into an add-on RPM which I have added to the Unidata AWIPS II release (not available to the public at this time, sorry).

Initial results are promising: the total time to ingest and decode the 0.5 degree GFS on CONDUIT (25k files, ~3.6 GB) was just over 1 hour.


I let this run for a few days to make sure the Qpid message queue remained active as in the past I've noticed high-volume grib message decoding would sometimes bottle-neck the system even though the dataflow through-put was low.  Here, it seems, the system can more than handle the 0.5/2.5 degree CONDUIT GFS.

Of course ingesting CONDUIT grids alone is one thing, ingesting them alongside point, satellite and radar data is another. That's the next step.

For more info on modifying the number of decoder threads, see: https://www.unidata.ucar.edu/staff/mjames/awips2/docs/threads.html

-Michael

Comments:

Post a Comment:
Comments are closed for this entry.
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« November 2022
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
8
9
10
11
12
13
15
16
17
18
19
20
21
23
24
25
26
27
29
30
   
       
Today