[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #ISJ-658184]: NIMAGE feed - big lags from idd.unidata starting ~16 UTC Monday August 5?

Hi Pete,

re:
> Here are the answers to the OS info questions from earlier today.
> 
> on idd-agg.aos.wisc.edu, the product queue is indeed on /dev/shm, and is 68 
> GiB
> -rw-r--r--. 1 ldm ldm 72159379456 Aug  8 21:23 /dev/shm/ldm.pq

OK.

re:
> The OS is CentOS 6.10. It's been at least a month since I updated any
> packages. It's running kernel 2.6.32-754.3.5.el6.centos.plus.x86_64

We are running the same kernel on all of our cluster backends, and it
seems to be newer than what you are running:

2.6.32-754.17.1.el6.x86_64

re:
> I have not installed any BIOS updates to address the intel processor bugs.

If you do updates using 'yum' then the firmware (microcode) updates are
included in the general updates.

re:
> The machine has 96 Gb of RAM total.

OK, thanks.

re:
> Let me know if you think of any other info that might be useful to see.

If you run 'iotop' (as 'root'), do you seen any ldmd processes having
excessively high values?  On lead, we were seeing at least one and
perhaps other ldmd processes reporting 99.9% values.  After moving
the LDM queue to RAM disk, we saw those numbers to drop to near zero.

Here is something really interesting:

Mike has been going through the real server backend machines for
our idd.unidata.ucar.edu cluster and moving their LDM queues to
RAM disk.  The result of doing this is well represented by the
latency plots for freshair2 machine at UWashington.  For
instance, here is the latency plot for CONDUIT and NGRID:

CONDUIT

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+freshair2.atmos.washington.edu

NGRID

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NGRID+freshair2.atmos.washington.edu

Given this dramatic drop in latencies for high volume feeds, it would be
very interesting to see what would happen if you switched your NIMAGE
feed from hera to idd.unidata.ucar.edu.  Are you game for the test?

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ISJ-658184
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.