[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[AWIPS #XCU-969788]: EDEX grib processing optimization question



Hi Pete,

Yes, this sounds normal. How many cpu's do you have on the system that you are 
running EDEX on? 

The script used to help calculate these numbers is in 
/awips2/edex/etc/ingestGrib.sh but it's dependent on the number of cpu's. 

The line you would want to look at is the "GRIB_MAX_GRID_POINTS" and compare 
how many grid points are set there and what your logs are complaining about. 
But with all of those models you still may continue to see a backup/latency of 
ingest around the major model times (0, 6, 12, 18Z)

You could also try increasing the GRIB_DECODE_THREADS.



> Greetings.
> 
> We are running RHEL-7 with the latest version of AWIPS2 and our system has 
> 256 GB mem.
> 
> As grib files arrive we submit them to EDEX via the 
> /awips2/ldm/dev/notifyAWIPS2-unidata.py script.
> 
> The suite of models we pull in are many including global GFS, Canadian, UK, 
> EC, ensemble means of EC, NAEFS, convective allowing models (conusarw, 
> conusmem2, wrfnssl, hwfv3, hrrr, nam3km) and two regional models (NAM and 
> Canadian).
> 
> We've noticed more than occasionally during high volume times (during on-hour 
> cycles like 12z or 00z data arrives vs off-hour 06 and 18z cycles) messages 
> in edex-ingestGrib-YYYYMMDD.log such as "Max Grids in memory for 
> Gridpersister exceeded. Waiting for grids to process".    And a significant 
> delay of an hour can develop as gribs arrive and get processed (and they all 
> do eventually get processed).
> 
> The "Grids in memory" message makes us think that there may be an adjustable 
> default setting that can be increased - is there such a setting you could 
> point us to adjust?
> 


Thanks,

Tiffany Meyer
AWIPS Lead Software Engineer
UCAR-Unidata

If you're interested, please feel free to fill out a survey about the support 
you receive:  
https://docs.google.com/forms/d/e/1FAIpQLSeDIkdk8qUMgq8ZdM4jhP-ubJPUOr-mJMQgxInwoAWoV5QcOw/viewform

Ticket Details
===================
Ticket ID: XCU-969788
Department: Support AWIPS
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.