On Sat, 11 Oct 2014, Phil Birnie wrote:
WARN: Processed oldest product in queue: 23704.5 s.
WARN: pbuf_flush(): write(15,,4096) to decoder took 10 s:
/home/ldm/decoders/dcgrib2 -d ldm/gempak/logs/dcgrib2_GFS.log -e
GEMTBL=/store1/gempak/GEMPAK6.7.0/gempak/tables
We never see either of these errors the rest of the week.
It's very strange - there's nothing else on the system that is running at that
particular time that would slow down the machine and it's been happening for
several months now. Both of these warnings would seem to indicate a lack of
resources, but I don't understand why this would occur on Saturdays and not
throughout the rest of the week.
Something is slowing down the machine---maybe a scour process?---causing
it to have an extremely long delay in writing products. Is there any way to
check the load average and other things to see what might be happening?
I have found that if the load average goes above 6 or so on my
older servers, it stops writing to disk and I get those errors.
I would check for:
Scouring times. Does anything just get scoured on that morning?
GEMPAK 6.7 has a grib decoder that can take up an entire CPU when
data is coming in (GEMPAK 7.1 takes care of that)
DNS outage? (Keep 8.8.8.8, Google's public DNS server on standby).
Those are off the top of my head...
Gilbert
*******************************************************************************
Gilbert Sebenste ********
(My opinions only!) ******
Staff Meteorologist, Northern Illinois University ****
E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx ***
web: http://weather.admin.niu.edu **
Twitter: http://www.twitter.com/NIU_Weather **
Facebook: http://www.facebook.com/niu.weather *
*******************************************************************************
