Another possible reason for this would be some resource starvation of that
ingest process:
* Excessive memory, swapping
* I/O bottlenecks (slow filesystem, large directory traversal w/some
filesystems)
* Insufficient threads in the pool for number of items to be processed
--
Erik
From: awips2-users-bounces@xxxxxxxxxxxxxxxx
[mailto:awips2-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Brian Bernard
Sent: Tuesday, March 29, 2016 9:19 AM
To: awips2-users@xxxxxxxxxxxxxxxx
Subject: [awips2-users] High Latency in Ingest Logs
Hi,
I've been checking the ingest logs, to see what has been causing a frustrating
problem where my ingestGrib stops after a couple of days, and noticed in many
of the observation products such as raob and synop that I have a very high
latency. Here is an example:
INFO 2016-03-29 12:57:10,095 [Ingest.sfcobs-1] Ingest:
EDEX: Ingest - sfcobs::
/awips2/data_store/maritime/20160329/12/SXUS23_KWNB_291200_469498449.2016032912
processed in: 2.2350 (sec) Latency: 50.6560 (sec)
INFO 2016-03-29 13:08:05,361 [Ingest.goessounding-1]
Ingest: EDEX: Ingest - goessounding::
/awips2/data_store/goessounding/JUTX06_KNES_291306_469516938.bufr.2016032913
processed in: 0.0030 (sec) Latency: 67.1950 (sec)
INFO 2016-03-29 13:08:06,210 [Ingest.goessounding-1] Ingest: EDEX: Ingest -
goessounding::
/awips2/data_store/goessounding/JUTX06_KNES_291306_469516939.bufr.2016032913
processed in: 0.0030 (sec) Latency: 68.0450 (sec)
INFO 2016-03-29 13:08:10,327 [Ingest.goessounding-1] Ingest: EDEX: Ingest -
goessounding::
/awips2/data_store/goessounding/JUTX06_KNES_291306_469516940.bufr.2016032913
processed in: 0.0030 (sec) Latency: 72.1610 (sec)
INFO 2016-03-29 13:08:14,203 [Ingest.goessounding-1] Ingest: EDEX: Ingest -
goessounding::
/awips2/data_store/goessounding/JUTX06_KNES_291306_469516941.bufr.2016032913
processed in: 0.0040 (sec) Latency: 76.0380 (sec)
>From presusing an older document that Michael James put together, the issue
>with high latency is due QPID. Should I re-install QPID?
Thanks,
Brian Bernard
________________________________
The information contained in this email message is intended only for the use of
the individual(s) to whom it is addressed and may contain information that is
privileged and sensitive. If you are not the intended recipient, or otherwise
have received this communication in error, please notify the sender immediately
by email at the above referenced address and note that any further
dissemination, distribution or copying of this communication is strictly
prohibited.
The U.S. Export Control Laws regulate the export and re-export of technology
originating in the United States. This includes the electronic transmission of
information and software to foreign countries and to certain foreign nationals.
Recipient agrees to abide by these laws and their regulations -- including the
U.S. Department of Commerce Export Administration Regulations and the U.S.
Department of State International Traffic in Arms Regulations -- and not to
transfer, by electronic transmission or otherwise, any content derived from
this email to either a foreign national or a foreign destination in violation
of such laws.
