AWIPS Tips: Ingesting GOES Satellite Data

AWIPS Tips

Welcome back to AWIPS Tips! In today’s blog, we’re going to describe how GOES satellite data is ingested by EDEX. If you ever need to troubleshoot GOES satellite products on your EDEX, you can use this blog to guide you through verifying the correct setup of the configuration files and learn other relevant information. We will start by detailing the initial components, then demonstrate an example.

GOES Product Source

GOES satellite products are in netCDF file format and distributed over NOAAPort as tiles that when combined make a full sector, such as CONUS or Full Disk.

Example full disk sector with 62 tiles
Example full disk sector with 62 tiles

At Unidata, before re-distributing the data to our users, we stitch the tiles together to produce one netCDF file.

GOES NetCDF Files

You can get quite a bit of information from just the name of the netCDF file.

Example filenames with corresponding properties
Example filenames with corresponding properties

In order for the data to be ingested by EDEX, configuration files must match the metadata and relate to what EDEX is expecting, most important are the Physical Element, Sector ID, and Creating Entity. We will reference these in a later step.

If you are noticing an error in your edex-ingest-satellite logfile (like below), you can investigate which of the configuration files are missing or need updating.

WARN 2022-07-08 16:25:03,345 9508 [Ingest.GOESR-2] GoesrNetcdfDecoder: No valid records were found in file: OR_ABI-L2-CMIPF-M6C13_G16_s20221882200216_e20221882209536_c20221882210036.nc

You can view the metadata of the raw file by running the ncdump command, which is installed with EDEX (and when the netcdf rpm is installed). Note: Because the netCDF output is large, you can pipe the output to more (see below) to show the output in a readable format.

ncdump OR_GLM-L2-GLMF-M6_G16_s20221891558000_e20221891559000_c20221891558300.nc | more

GOESR EDEX Plugin

There are two types of configurations used by the GOESR edex plugin depending on the products you are working with:

Single Channel (Channel 1-16)

/awips2/edex/data/utility/common_static/base/satellite/goesr/descriptions/Sectorized_CMI

Derived Products (ex. GLM, Cloud Mask, Sea Surface Temp, etc.)

/awips2/edex/data/utility/common_static/base/satellite/goesr/descriptions/Level2

NOTE: While GLM Flash Point data and Derived Motion Winds are GOES products, they are decoded by the dmw plugin, not the GOESR edex plugin.

Example

Let’s now take a closer look at an example of GLM data.

OR_GLM-L2-GLMF-M6_G16_s20221891558000_e20221891559000_c20221891558300.nc

We know from the filename that we are working with GOES-East data, so we’ll start by looking at the GOES-East-Sectors.xml configuration file.

We need to find the sectorid for the data, which is a property that EDEX needs for proper configuration. To find the sectorid, we match the orbital_slot and scene_id values in the netCDF file to the values in GOES-East-Sectors.xml.

NOTE: When working with GOES-West data, use GOES-West-Sectors.xml

Using ncdump, locate the dataset_name, orbital_slot, and scene_id values in the metadata. For example:

ncdump OR_GLM-L2-GLMF-M6_G16_s20221891558000_e20221891559000_c20221891558300.nc | grep 'dataset_name\|orbital_slot\|scene_id'

Example ncdump output
Example ncdump output

common sector configurations
A few common sector configurations (this is not an exhaustive list)

These sector configurations are used for all derived level2 products.

Next, we will look at a product description file, specific to GLM

/awips2/edex/data/utility/common_static/base/satellite/goesr/descriptions/Level2/GLMG_FD_tile.xml

to decode the different variables in the GLM file. We can look at the metadata to see which variables are included by using ncdump again:

ncdump OR_GLM-L2-GLMF-M6_G16_s20221881848000_e20221881849000_c20221881848300.nc | more

Example variable metadata
Example variable metadata
Click to enlarge

After the X and Y coordinates, the specific GLM products follow; each variable needs an entry in the configuration file. For example, below is the associated configuration for the Total_Optical_energy variable. The variable name and units are found in the metadata for the netCDF file. (Note that not every variable will have units.) There is also a discreteMask variable in this configuration. If the value is set to 0, then data will be displayed when loaded for each gridpoint, otherwise it will be masked and not displayed. The physicalElement is then set to a value, this is the value EDEX stores the processed data as.

Example variable configuration
Example variable configuration for Total_Optical_Energy

After Configuration

When referencing how to add new products to a menu, where to find them in the product browser, or access them via python-awips, you’ll need to know the:

  • creatingentity
  • sectorid
  • physicalelement

CAVE Product Browser
CAVE Product Browser

In the event you have additional questions about ingesting GOES products into EDEX, you can reach out to us at support-awips@unidata.ucar.edu. And check back in two weeks for the next blog post, where we will share options for visualizing Watches, Warnings, and Advisories in CAVE.

To view archived blogs, visit the AWIPS Tips blog tag, and get notified of the latest updates from the AWIPS team by signing up for the AWIPS mailing list. Questions or suggestions for the team on future topics? Let us know at support-awips@unidata.ucar.edu

Comments:

Post a Comment:
  • HTML Syntax: Allowed
News@Unidata
News and information from the Unidata Program Center
News@Unidata
News and information from the Unidata Program Center

Welcome

FAQs

Developers’ blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« August 2022
SunMonTueWedThuFriSat
 
4
5
6
7
9
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
   
       
Today