[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[LDM #GXL-914371]: Strange LDM queue behavior - 6.11.6
- Subject: [LDM #GXL-914371]: Strange LDM queue behavior - 6.11.6
- Date: Thu, 16 Jan 2014 09:41:02 -0700
Ryan,
The LDM product-queue is designed to delete products from the product-queue in
order to make room for new products. Eligible products are removed in FIFO
order. Products that are being processed (by pqact(1), for example) are not
elegible are their deletion is skipped.
At about 2014-01-15T17:28:04, the age of the oldest product in the queue starts
increasing at the rate of one second per second, while the number of products
in the queue continues to fluctuate. I suspect that the oldest product in the
queue is stuck because it's being processed by a pqact(1) process that is
waiting on a decoder to finish reading a product from standard input.
Look for a hung pqact(1) process (one that's not accumulating any CPU time).
Figure-out what decoder it's piping the product to and fix the decoder.
An alternative explanation is that a pqact(1) FILE action (rather than a PIPE
action) is hung. This could happen if, for example, a product was being written
to an NFS-mounted disk and the disk was somehow unavailable. This possibility
is less likely in my opinion.
> We're running LDM 6.11.6 on RHEL 5.10. Usually our queue retention time is
> around 2000-3000 for this system. I noticed this morning that our product
> age is now about 83000. Any ideas what would cause this? I assume
> there's a product that's not being scoured out of the queue or something.
> I've been running a pqmon every 5 min for a few days, the output is below.
>
> Jan 15 09:08:04 pqmon NOTE: 68880 2 131118 2999122344 199999
> 3 0 861416 1223
> Jan 15 09:13:04 pqmon NOTE: 55024 1 144975 2999994920 199999
> 3 0 5592 1180
> Jan 15 09:18:04 pqmon NOTE: 54444 1 145555 2999780712 199999
> 3 0 219800 1378
> Jan 15 09:23:04 pqmon NOTE: 61308 3 138689 2998617112 199999
> 3 0 1381056 1567
> Jan 15 09:28:04 pqmon NOTE: 68894 1 131105 2998004112 199999
> 3 0 1996400 1783
> Jan 15 09:33:04 pqmon NOTE: 77441 2 122557 2999720288 199999
> 3 0 279816 1984
> Jan 15 09:38:04 pqmon NOTE: 84783 1 115216 2999973840 199999
> 3 0 26672 2154
> Jan 15 09:43:04 pqmon NOTE: 90265 2 109733 2999056584 199999
> 3 0 819888 2311
> Jan 15 09:48:04 pqmon NOTE: 98532 1 101467 2997856288 199999
> 3 0 2144224 2555
> Jan 15 09:53:04 pqmon NOTE: 106470 2 93528 2999674784 199999
> 3 0 315192 2750
> Jan 15 09:58:04 pqmon NOTE: 115677 1 84322 2999989072 199999
> 3 0 11440 2959
> Jan 15 10:03:04 pqmon NOTE: 119168 1 80831 2999207560 199999
> 3 0 792952 3036
> Jan 15 10:08:04 pqmon NOTE: 122021 1 77978 2999993432 199999
> 3 0 7080 3060
> Jan 15 10:13:04 pqmon NOTE: 121645 2 78353 2999913648 199999
> 3 0 86560 3031
> Jan 15 10:18:04 pqmon NOTE: 128255 1 71744 2999989448 199999
> 3 0 11064 3032
> Jan 15 10:23:04 pqmon NOTE: 128989 1 71010 2994736184 199999
> 3 0 5264328 3073
> Jan 15 10:28:04 pqmon NOTE: 126473 1 73526 2999987928 199999
> 3 0 12584 3068
> Jan 15 10:33:04 pqmon NOTE: 124818 1 75181 2999545496 199999
> 3 0 455016 3045
> Jan 15 10:38:04 pqmon NOTE: 132622 1 67377 2999710512 199999
> 3 0 290000 3072
> Jan 15 10:43:04 pqmon NOTE: 127110 2 72888 2999746560 199999
> 3 0 250392 2961
> Jan 15 10:48:04 pqmon NOTE: 130480 1 69519 2999924272 199999
> 3 0 76240 3108
> Jan 15 10:53:04 pqmon NOTE: 135467 2 64531 2999750264 199999
> 3 0 250040 3302
> Jan 15 10:58:04 pqmon NOTE: 135531 2 64467 2999192032 199999
> 3 0 783136 3312
> Jan 15 11:03:04 pqmon NOTE: 129828 2 70170 2999942144 199999
> 3 0 54440 3196
> Jan 15 11:08:04 pqmon NOTE: 118541 1 81458 2996797888 199999
> 3 0 3202624 3085
> Jan 15 11:13:04 pqmon NOTE: 114348 1 85651 2999986440 199999
> 3 0 14072 2996
> Jan 15 11:18:04 pqmon NOTE: 112982 1 87017 2999970768 199999
> 3 0 29744 2943
> Jan 15 11:23:04 pqmon NOTE: 105589 1 94410 2999992344 199999
> 3 0 8168 2833
> Jan 15 11:28:04 pqmon NOTE: 105142 1 94857 2999112248 199999
> 3 0 888264 2851
> Jan 15 11:33:04 pqmon NOTE: 110261 1 89738 2999520792 199999
> 3 0 479720 3017
> Jan 15 11:38:04 pqmon NOTE: 99409 2 100589 2999650288 199999
> 3 0 327272 2664
> Jan 15 11:43:04 pqmon NOTE: 101993 1 98006 2999994696 199999
> 3 0 5816 2756
> Jan 15 11:48:04 pqmon NOTE: 98880 1 101119 2994070432 199999
> 3 0 5930080 2737
> Jan 15 11:53:04 pqmon NOTE: 102255 2 97743 2999998792 199999
> 3 0 1328 2863
> Jan 15 11:58:04 pqmon NOTE: 106212 1 93787 2999560560 199999
> 3 0 439952 2957
> Jan 15 12:03:04 pqmon NOTE: 113163 2 86835 2999649544 199999
> 3 0 350416 3144
> Jan 15 12:08:04 pqmon NOTE: 111220 2 88778 2999996288 199999
> 3 0 3320 3042
> Jan 15 12:13:04 pqmon NOTE: 112620 1 87379 2999932536 199999
> 3 0 67976 3071
> Jan 15 12:18:04 pqmon NOTE: 116997 1 83002 2999994240 199999
> 3 0 6272 3201
> Jan 15 12:23:04 pqmon NOTE: 119552 1 80447 2999975224 199999
> 3 0 25288 3252
> Jan 15 12:28:04 pqmon NOTE: 120367 1 79632 2999869752 199999
> 3 0 130760 3251
> Jan 15 12:33:04 pqmon NOTE: 122922 1 77077 2999739160 199999
> 3 0 261352 3330
> Jan 15 12:38:04 pqmon NOTE: 127240 2 72758 2999438992 199999
> 3 0 557880 3423
> Jan 15 12:43:04 pqmon NOTE: 125748 2 74250 2999233376 199999
> 3 0 764792 3384
> Jan 15 12:48:04 pqmon NOTE: 122368 2 77630 2999880184 199999
> 3 0 119104 3262
> Jan 15 12:53:04 pqmon NOTE: 121420 1 78579 2999921296 199999
> 3 0 79216 3278
> Jan 15 12:58:04 pqmon NOTE: 113932 1 86067 2999232656 199999
> 3 0 767856 3099
> Jan 15 13:03:04 pqmon NOTE: 110314 2 89684 2999923560 199999
> 3 0 76584 3048
> Jan 15 13:08:04 pqmon NOTE: 110522 2 89476 2998873440 199999
> 3 0 1126776 3054
> Jan 15 13:13:04 pqmon NOTE: 108993 2 91005 2999755224 199999
> 3 0 229864 3035
> Jan 15 13:18:04 pqmon NOTE: 106514 1 93485 2999996656 199999
> 3 0 3856 2978
> Jan 15 13:23:04 pqmon NOTE: 104838 2 95160 2999997288 199999
> 3 0 1728 2918
> Jan 15 13:28:04 pqmon NOTE: 106363 1 93636 2999998832 199999
> 3 0 1680 2891
> Jan 15 13:33:04 pqmon NOTE: 104246 1 95753 2997189176 199999
> 3 0 2811336 2765
> Jan 15 13:38:04 pqmon NOTE: 105130 1 94869 2990797880 199999
> 3 0 9202632 2732
> Jan 15 13:43:04 pqmon NOTE: 110527 2 89471 2999346392 199999
> 3 0 651960 2851
> Jan 15 13:48:04 pqmon NOTE: 111165 1 88834 2999791976 199999
> 3 0 208536 2833
> Jan 15 13:53:04 pqmon NOTE: 117811 2 82187 2999355152 199999
> 3 0 644040 3008
> Jan 15 13:58:04 pqmon NOTE: 113827 2 86171 2997953408 199999
> 3 0 2044088 2852
> Jan 15 14:03:04 pqmon NOTE: 115986 2 84012 2999998880 199999
> 3 0 1304 2863
> Jan 15 14:08:04 pqmon NOTE: 123373 1 76626 3000000160 199999
> 3 0 352 3044
> Jan 15 14:13:04 pqmon NOTE: 122248 1 77751 2999920640 199999
> 3 0 79872 2994
> Jan 15 14:18:04 pqmon NOTE: 129551 1 70448 2999895544 199999
> 3 0 104968 3194
> Jan 15 14:23:04 pqmon NOTE: 122187 2 77811 2999575472 199999
> 3 0 357024 3025
> Jan 15 14:28:04 pqmon NOTE: 110758 1 89241 2999944960 199999
> 3 0 55552 2754
> Jan 15 14:33:04 pqmon NOTE: 110561 2 89437 2998889440 199999
> 3 0 1109416 2746
> Jan 15 14:38:04 pqmon NOTE: 104651 2 95347 2999178096 199999
> 3 0 520832 2630
> Jan 15 14:43:04 pqmon NOTE: 108676 2 91322 2999623920 199999
> 3 0 346408 2799
> Jan 15 14:48:04 pqmon NOTE: 101841 1 98158 3000000096 199999
> 3 0 416 2676
> Jan 15 14:53:04 pqmon NOTE: 114836 2 85162 2998919640 199999
> 3 0 705176 2516
> Jan 15 14:58:04 pqmon NOTE: 106137 1 93862 2999344392 199999
> 3 0 656120 2280
> Jan 15 15:03:04 pqmon NOTE: 110189 2 89809 2999970552 199999
> 3 0 22632 2413
> Jan 15 15:08:04 pqmon NOTE: 115260 1 84739 2999329032 199999
> 3 0 671480 2565
> Jan 15 15:13:04 pqmon NOTE: 117627 2 82371 2987104840 199999
> 3 0 12243816 2641
> Jan 15 15:18:04 pqmon NOTE: 119851 1 80148 2997499632 199999
> 3 0 2500880 2712
> Jan 15 15:23:04 pqmon NOTE: 119855 2 80143 2999875000 199999
> 3 0 79992 2681
> Jan 15 15:28:04 pqmon NOTE: 122140 2 77858 2999780784 199999
> 3 0 219048 2690
> Jan 15 15:33:04 pqmon NOTE: 127669 1 72330 2999304608 199999
> 3 0 695904 2795
> Jan 15 15:38:04 pqmon NOTE: 136797 2 63201 2999991384 199999
> 3 0 8896 3039
> Jan 15 15:43:04 pqmon NOTE: 134651 1 65348 2999997992 199999
> 3 0 2520 3164
> Jan 15 15:48:04 pqmon NOTE: 128520 1 71479 2999996560 199999
> 3 0 3952 3314
> Jan 15 15:53:04 pqmon NOTE: 136632 1 63367 2999940688 199999
> 3 0 59824 3548
> Jan 15 15:58:04 pqmon NOTE: 140775 1 59224 2999788216 199999
> 3 0 212296 3622
> Jan 15 16:03:04 pqmon NOTE: 140951 1 59048 2999081632 199999
> 3 0 918880 3613
> Jan 15 16:08:04 pqmon NOTE: 152188 1 47811 2999135712 199999
> 3 0 864800 3757
> Jan 15 16:13:04 pqmon NOTE: 158953 2 41045 2999933752 199999
> 3 0 66472 3892
> Jan 15 16:18:04 pqmon NOTE: 161553 1 38446 2987672208 199999
> 3 0 12328304 3957
> Jan 15 16:23:04 pqmon NOTE: 160932 1 39067 2999996392 199999
> 3 0 4120 3951
> Jan 15 16:28:04 pqmon NOTE: 163008 1 36991 2999943072 199999
> 3 0 57440 3870
> Jan 15 16:33:04 pqmon NOTE: 161850 2 38148 2999966600 199999
> 3 0 33720 3876
> Jan 15 16:38:04 pqmon NOTE: 166190 1 33809 2999299168 199999
> 3 0 701344 3996
> Jan 15 16:43:04 pqmon NOTE: 162019 1 37980 2999950352 199999
> 3 0 50160 3947
> Jan 15 16:48:04 pqmon NOTE: 160208 1 39791 2999280456 199999
> 3 0 720056 4034
> Jan 15 16:53:04 pqmon NOTE: 165492 1 34507 2999942528 199999
> 3 0 57984 4202
> Jan 15 16:58:04 pqmon NOTE: 157311 2 42687 2999941136 199999
> 3 0 33560 3934
> Jan 15 17:03:04 pqmon NOTE: 158121 1 41878 2991983080 199999
> 3 0 8017432 3985
> Jan 15 17:08:04 pqmon NOTE: 161494 1 38505 2999981824 199999
> 3 0 18688 4109
> Jan 15 17:13:04 pqmon NOTE: 164410 1 35589 2996538456 199999
> 3 0 3462056 4212
> Jan 15 17:18:04 pqmon NOTE: 162553 1 37446 2999996232 199999
> 3 0 4280 4277
> Jan 15 17:23:04 pqmon NOTE: 153369 1 46630 2999987688 199999
> 3 0 12824 4135
> Jan 15 17:28:04 pqmon NOTE: 149446 1 50553 2999815304 199999
> 3 0 185208 4022
> Jan 15 17:33:04 pqmon NOTE: 150864 2 49134 2990187640 199999
> 3 0 9812464 4110
> Jan 15 17:38:04 pqmon NOTE: 141671 1 58328 2994037944 199999
> 3 0 5962568 4102
> Jan 15 17:43:04 pqmon NOTE: 132960 1 67039 2992575384 199999
> 3 0 7425128 4402
> Jan 15 17:48:04 pqmon NOTE: 130732 1 69267 2999442320 199999
> 3 0 558192 4702
> Jan 15 17:53:04 pqmon NOTE: 128911 2 71087 2999426528 199999
> 3 0 543232 5002
> Jan 15 17:58:04 pqmon NOTE: 128692 2 71306 2999973944 199999
> 3 0 23728 5302
> Jan 15 18:03:04 pqmon NOTE: 117698 1 82301 2999886408 199999
> 3 0 114104 5602
> Jan 15 18:08:04 pqmon NOTE: 115135 1 84864 2999935096 199999
> 3 0 65416 5902
> Jan 15 18:13:04 pqmon NOTE: 114673 2 85325 2999929440 199999
> 3 0 69128 6202
> Jan 15 18:18:04 pqmon NOTE: 119792 1 80207 2999438136 199999
> 3 0 562376 6502
> Jan 15 18:23:04 pqmon NOTE: 123103 2 76895 2999854392 199999
> 3 0 102696 6802
> Jan 15 18:28:04 pqmon NOTE: 121993 1 78006 2999077648 199999
> 3 0 922864 7102
> Jan 15 18:33:04 pqmon NOTE: 122147 2 77851 2999684704 199999
> 3 0 305064 7402
> Jan 15 18:38:04 pqmon NOTE: 116716 2 83282 2995443440 199999
> 3 0 4556600 7702
> Jan 15 18:43:04 pqmon NOTE: 103193 1 96806 2999999344 199999
> 3 0 1168 8002
> Jan 15 18:48:04 pqmon NOTE: 104729 1 95270 2999335208 199999
> 3 0 665304 8302
> Jan 15 18:53:04 pqmon NOTE: 111790 3 88207 2998877768 199999
> 3 0 806472 8602
> Jan 15 18:58:04 pqmon NOTE: 108890 1 91109 2986636600 199999
> 3 0 13363912 8902
> Jan 15 19:03:04 pqmon NOTE: 113426 2 86572 2999985008 199999
> 3 0 14656 9202
> Jan 15 19:08:04 pqmon NOTE: 104617 1 95382 2999997920 199999
> 3 0 2592 9502
> Jan 15 19:13:04 pqmon NOTE: 104698 2 95300 2999836568 199999
> 3 0 115760 9802
> Jan 15 19:18:04 pqmon NOTE: 104097 1 95902 2999998744 199999
> 3 0 1768 10102
> Jan 15 19:23:04 pqmon NOTE: 103259 2 96739 2999590472 199999
> 3 0 404680 10402
> Jan 15 19:28:04 pqmon NOTE: 101465 2 98533 2999995680 199999
> 3 0 2928 10702
> Jan 15 19:33:04 pqmon NOTE: 105548 1 94451 2999773320 199999
> 3 0 227192 11002
> Jan 15 19:38:04 pqmon NOTE: 106022 1 93977 2999193360 199999
> 3 0 807152 11302
> Jan 15 19:43:04 pqmon NOTE: 98425 1 101574 2995830112 199999
> 3 0 4170400 11602
> Jan 15 19:48:04 pqmon NOTE: 93846 1 106153 2999412800 199999
> 3 0 587712 11902
> Jan 15 19:53:04 pqmon NOTE: 97186 1 102813 2999989928 199999
> 3 0 10584 12202
> Jan 15 19:58:04 pqmon NOTE: 93350 2 106648 2998100720 199999
> 3 0 1651104 12502
> Jan 15 20:03:04 pqmon NOTE: 92681 1 107318 2999996496 199999
> 3 0 4016 12802
> Jan 15 20:08:04 pqmon NOTE: 96325 2 103673 2999995840 199999
> 3 0 2336 13102
> Jan 15 20:13:04 pqmon NOTE: 90580 1 109419 2999992160 199999
> 3 0 8352 13402
> Jan 15 20:18:04 pqmon NOTE: 91078 1 108921 2999982224 199999
> 3 0 18288 13702
> Jan 15 20:23:04 pqmon NOTE: 97475 1 102524 2998476896 199999
> 3 0 1523616 14002
> Jan 15 20:28:04 pqmon NOTE: 98080 2 101918 2999541280 199999
> 3 0 422984 14302
> Jan 15 20:33:04 pqmon NOTE: 100597 1 99402 2999067240 199999
> 3 0 933272 14602
> Jan 15 20:38:04 pqmon NOTE: 108308 1 91691 2983436232 199999
> 3 0 16564280 14902
> Jan 15 20:43:04 pqmon NOTE: 107329 1 92670 2999100976 199999
> 3 0 899536 15202
> Jan 15 20:48:04 pqmon NOTE: 115074 1 84925 2996470824 199999
> 3 0 3529688 15502
> Jan 15 20:53:04 pqmon NOTE: 132062 1 67937 2999999112 199999
> 3 0 1400 15802
> Jan 15 20:58:04 pqmon NOTE: 128502 1 71497 2999390240 199999
> 3 0 610272 16102
> Jan 15 21:03:04 pqmon NOTE: 131351 1 68648 2988527008 199999
> 3 0 11473504 16402
> Jan 15 21:08:04 pqmon NOTE: 129926 1 70073 2998905968 199999
> 3 0 1094544 16702
> Jan 15 21:13:04 pqmon NOTE: 130613 2 69385 2999992216 199999
> 3 0 6616 17002
> Jan 15 21:18:04 pqmon NOTE: 133114 1 66885 2989062680 199999
> 3 0 10937832 17302
> Jan 15 21:23:04 pqmon NOTE: 129279 1 70720 2988288368 199999
> 3 0 11712144 17602
> Jan 15 21:28:04 pqmon NOTE: 122838 2 77160 2999707552 199999
> 3 0 286264 17902
> Jan 15 21:33:04 pqmon NOTE: 110772 1 89227 2999995416 199999
> 3 0 5096 18202
> Jan 15 21:38:04 pqmon NOTE: 106218 1 93781 2999996160 199999
> 3 0 4352 18502
> Jan 15 21:43:04 pqmon NOTE: 106253 1 93746 2999115000 199999
> 3 0 885512 18802
> Jan 15 21:48:04 pqmon NOTE: 104549 1 95450 2999811344 199999
> 3 0 189168 19102
> Jan 15 21:53:04 pqmon NOTE: 106494 1 93505 2999476176 199999
> 3 0 524336 19402
> Jan 15 21:58:04 pqmon NOTE: 100023 2 99975 2999845032 199999
> 3 0 122728 19702
> Jan 15 22:03:04 pqmon NOTE: 103816 3 96181 2999954216 199999
> 3 0 32928 20002
> Jan 15 22:08:04 pqmon NOTE: 109079 3 90918 2999930760 199999
> 3 0 62776 20302
> Jan 15 22:13:05 pqmon NOTE: 114698 1 85301 2993996000 199999
> 3 0 6004512 20602
> Jan 15 22:18:05 pqmon NOTE: 104681 1 95318 2999978520 199999
> 3 0 21992 20902
> Jan 15 22:23:05 pqmon NOTE: 105218 1 94781 2985611776 199999
> 3 0 14388736 21202
> Jan 15 22:28:05 pqmon NOTE: 117269 2 82729 2998557480 199999
> 3 0 1426792 21502
> Jan 15 22:33:05 pqmon NOTE: 120017 1 79982 2999991200 199999
> 3 0 9312 21802
> Jan 15 22:38:05 pqmon NOTE: 129011 1 70988 2999677392 199999
> 3 0 323120 22102
> Jan 15 22:43:05 pqmon NOTE: 130091 1 69908 2999985976 199999
> 3 0 14536 22402
> Jan 15 22:48:05 pqmon NOTE: 132076 1 67923 2999663848 199999
> 3 0 336664 22702
> Jan 15 22:53:05 pqmon NOTE: 134653 1 65346 2999996240 199999
> 3 0 4272 23002
> Jan 15 22:58:05 pqmon NOTE: 130713 1 69286 2999340400 199999
> 3 0 660112 23302
> Jan 15 23:03:05 pqmon NOTE: 122173 1 77826 2999368120 199999
> 3 0 632392 23602
> Jan 15 23:08:05 pqmon NOTE: 117696 1 82303 2999925168 199999
> 3 0 75344 23902
> Jan 15 23:13:05 pqmon NOTE: 119009 1 80990 2999826272 199999
> 3 0 174240 24202
> Jan 15 23:18:05 pqmon NOTE: 122917 1 77082 2982466784 199999
> 3 0 17533728 24502
> Jan 15 23:23:05 pqmon NOTE: 100131 1 99868 2999988928 199999
> 3 0 11584 24802
> Jan 15 23:28:05 pqmon NOTE: 91664 1 108335 2999978368 199999
> 3 0 22144 25102
> Jan 15 23:33:05 pqmon NOTE: 89866 1 110133 2999277296 199999
> 3 0 723216 25402
> Jan 15 23:38:05 pqmon NOTE: 96038 1 103961 2982654512 199999
> 3 0 17346000 25702
> Jan 15 23:43:05 pqmon NOTE: 97076 2 102922 2999921360 199999
> 3 0 62768 26002
> Jan 15 23:48:05 pqmon NOTE: 99106 1 100893 2999988920 199999
> 3 0 11592 26302
> Jan 15 23:53:05 pqmon NOTE: 98860 2 101138 2999964976 199999
> 3 0 35344 26602
> Jan 15 23:58:05 pqmon NOTE: 89755 1 110244 2999880752 199999
> 3 0 119760 26902
> Jan 16 00:03:05 pqmon NOTE: 91765 1 108234 2992141912 199999
> 3 0 7858600 27202
> Jan 16 00:08:05 pqmon NOTE: 99651 2 100347 2999015632 199999
> 3 0 793232 27502
> Jan 16 00:13:05 pqmon NOTE: 107222 1 92777 2999807680 199999
> 3 0 192832 27802
> Jan 16 00:18:05 pqmon NOTE: 115100 1 84899 2987289648 199999
> 3 0 12710864 28102
> Jan 16 00:23:05 pqmon NOTE: 115670 2 84328 2999962496 199999
> 3 0 37800 28402
> Jan 16 00:28:05 pqmon NOTE: 110806 1 89193 2998697688 199999
> 3 0 1302824 28702
> Jan 16 00:33:05 pqmon NOTE: 116684 3 83313 2999004208 199999
> 3 0 903680 29002
> Jan 16 00:38:05 pqmon NOTE: 118185 1 81814 2997243824 199999
> 3 0 2756688 29302
> Jan 16 00:43:05 pqmon NOTE: 112997 2 87001 2999352864 199999
> 3 0 647456 29602
> Jan 16 00:48:05 pqmon NOTE: 113986 1 86013 2999298256 199999
> 3 0 702256 29902
> Jan 16 00:53:05 pqmon NOTE: 113483 1 86516 2998141440 199999
> 3 0 1859072 30202
> Jan 16 00:58:05 pqmon NOTE: 111311 1 88688 2999996280 199999
> 3 0 4232 30502
> Jan 16 01:03:05 pqmon NOTE: 109354 2 90644 2999980528 199999
> 3 0 18384 30802
> Jan 16 01:08:05 pqmon NOTE: 99753 2 100245 2999827456 199999
> 3 0 170344 31102
> Jan 16 01:13:05 pqmon NOTE: 97838 3 102159 2999759208 199999
> 3 0 105464 31402
> Jan 16 01:18:05 pqmon NOTE: 93491 2 106507 2999331112 199999
> 3 0 669016 31702
> Jan 16 01:23:05 pqmon NOTE: 95106 2 104892 2999079864 199999
> 3 0 885584 32002
> Jan 16 01:28:05 pqmon NOTE: 99128 2 100870 2999810472 199999
> 3 0 189504 32302
> Jan 16 01:33:05 pqmon NOTE: 98628 3 101369 2998785312 199999
> 3 0 974576 32602
> Jan 16 01:38:05 pqmon NOTE: 95143 1 104856 2999549824 199999
> 3 0 450688 32902
> Jan 16 01:43:05 pqmon NOTE: 93997 1 106002 2998081784 199999
> 3 0 1918728 33202
> Jan 16 01:48:05 pqmon NOTE: 100976 1 99023 2986613832 199999
> 3 0 13386680 33502
> Jan 16 01:53:05 pqmon NOTE: 105482 1 94517 2999830160 199999
> 3 0 170352 33802
> Jan 16 01:58:05 pqmon NOTE: 107343 2 92655 2999963024 199999
> 3 0 33816 34102
> Jan 16 02:03:05 pqmon NOTE: 110823 1 89176 2999800696 199999
> 3 0 199816 34402
> Jan 16 02:08:05 pqmon NOTE: 111959 1 88040 2999951936 199999
> 3 0 48576 34702
> Jan 16 02:13:05 pqmon NOTE: 107900 2 92098 2999911832 199999
> 3 0 73056 35002
> Jan 16 02:18:05 pqmon NOTE: 108927 1 91072 2998169936 199999
> 3 0 1830576 35302
> Jan 16 02:23:05 pqmon NOTE: 110916 1 89083 2999964528 199999
> 3 0 35984 35602
> Jan 16 02:28:05 pqmon NOTE: 110144 1 89855 2999990840 199999
> 3 0 9672 35902
> Jan 16 02:33:05 pqmon NOTE: 96957 2 103041 2999767064 199999
> 3 0 198792 36202
> Jan 16 02:38:05 pqmon NOTE: 102963 2 97035 2999986296 199999
> 3 0 13992 36502
> Jan 16 02:43:05 pqmon NOTE: 98896 2 101102 2999653344 199999
> 3 0 323848 36802
> Jan 16 02:48:05 pqmon NOTE: 94976 1 105023 2999416584 199999
> 3 0 583928 37102
> Jan 16 02:53:05 pqmon NOTE: 113625 2 86373 2999769864 199999
> 3 0 206232 37402
> Jan 16 02:58:05 pqmon NOTE: 113852 1 86147 2998730080 199999
> 3 0 1270432 37702
> Jan 16 03:03:05 pqmon NOTE: 115495 1 84504 2999990496 199999
> 3 0 10016 38002
> Jan 16 03:08:05 pqmon NOTE: 117448 1 82551 2985156312 199999
> 3 0 14844200 38302
> Jan 16 03:13:05 pqmon NOTE: 113020 1 86979 2997540336 199999
> 3 0 2460176 38602
> Jan 16 03:18:05 pqmon NOTE: 119349 2 80649 2999999528 199999
> 3 0 776 38902
> Jan 16 03:23:05 pqmon NOTE: 120601 2 79397 2999908712 199999
> 3 0 90840 39202
> Jan 16 03:28:05 pqmon NOTE: 125270 2 74728 2999082600 199999
> 3 0 917584 39502
> Jan 16 03:33:05 pqmon NOTE: 132394 1 67605 2999998584 199999
> 3 0 1928 39802
> Jan 16 03:38:05 pqmon NOTE: 141688 2 58310 2999992728 199999
> 3 0 6088 40102
> Jan 16 03:43:05 pqmon NOTE: 135480 1 64519 2999264864 199999
> 3 0 735648 40402
> Jan 16 03:48:05 pqmon NOTE: 136152 1 63847 2999998024 199999
> 3 0 2488 40702
> Jan 16 03:53:05 pqmon NOTE: 138600 2 61398 2999894280 199999
> 3 0 106040 41002
> Jan 16 03:58:05 pqmon NOTE: 142133 2 57865 2989110544 199999
> 3 0 8752000 41302
> Jan 16 04:03:05 pqmon NOTE: 143839 2 56159 2999900752 199999
> 3 0 97256 41602
> Jan 16 04:08:05 pqmon NOTE: 152826 2 47172 2999211496 199999
> 3 0 766528 41902
> Jan 16 04:13:05 pqmon NOTE: 160407 1 39592 2999991936 199999
> 3 0 8576 42202
> Jan 16 04:18:05 pqmon NOTE: 166602 2 33396 2998493472 199999
> 3 0 1504984 42502
> Jan 16 04:23:05 pqmon NOTE: 169154 1 30845 2999949696 199999
> 3 0 50816 42802
> Jan 16 04:28:05 pqmon NOTE: 171322 1 28677 2999995824 199999
> 3 0 4688 43102
> Jan 16 04:33:05 pqmon NOTE: 168886 2 31112 2999917064 199999
> 3 0 81904 43402
> Jan 16 04:38:05 pqmon NOTE: 172142 1 27857 2999983784 199999
> 3 0 16728 43702
> Jan 16 04:43:05 pqmon NOTE: 162336 2 37662 2999683904 199999
> 3 0 313600 44002
> Jan 16 04:48:05 pqmon NOTE: 162987 1 37012 2999947096 199999
> 3 0 53416 44302
> Jan 16 04:53:05 pqmon NOTE: 160293 1 39706 2999980416 199999
> 3 0 20096 44602
> Jan 16 04:58:05 pqmon NOTE: 154319 1 45680 2999994616 199999
> 3 0 5896 44902
> Jan 16 05:03:05 pqmon NOTE: 147510 1 52489 2998331104 199999
> 3 0 1669408 45202
> Jan 16 05:08:05 pqmon NOTE: 132937 2 67061 2999757280 199999
> 3 0 202664 45502
> Jan 16 05:13:05 pqmon NOTE: 129357 1 70642 2996257240 199999
> 3 0 3743272 45802
> Jan 16 05:18:05 pqmon NOTE: 122919 1 77080 2998928256 199999
> 3 0 1072256 46102
> Jan 16 05:23:05 pqmon NOTE: 116004 1 83995 2999992712 199999
> 3 0 7800 46402
> Jan 16 05:28:05 pqmon NOTE: 112696 1 87303 2999987416 199999
> 3 0 13096 46702
> Jan 16 05:33:05 pqmon NOTE: 112741 1 87258 2999352944 199999
> 3 0 647568 47002
> Jan 16 05:38:05 pqmon NOTE: 111395 1 88604 2999310992 199999
> 3 0 689520 47302
> Jan 16 05:43:05 pqmon NOTE: 110392 1 89607 2999492744 199999
> 3 0 507768 47602
> Jan 16 05:48:05 pqmon NOTE: 104553 1 95446 2999795472 199999
> 3 0 205040 47902
> Jan 16 05:53:05 pqmon NOTE: 115640 1 84359 2998223776 199999
> 3 0 1776736 48202
> Jan 16 05:58:05 pqmon NOTE: 124619 1 75380 2999976216 199999
> 3 0 24296 48502
> Jan 16 06:03:05 pqmon NOTE: 131891 1 68108 2998730856 199999
> 3 0 1269656 48802
> Jan 16 06:08:05 pqmon NOTE: 125340 1 74659 2999916480 199999
> 3 0 84032 49102
> Jan 16 06:13:05 pqmon NOTE: 115240 1 84759 2999048872 199999
> 3 0 951640 49402
> Jan 16 06:18:05 pqmon NOTE: 115878 1 84121 2999034040 199999
> 3 0 966472 49702
> Jan 16 06:23:05 pqmon NOTE: 120552 1 79447 2985583128 199999
> 3 0 14417384 50002
> Jan 16 06:28:05 pqmon NOTE: 125155 1 74844 2999199200 199999
> 3 0 801312 50302
> Jan 16 06:33:05 pqmon NOTE: 129198 1 70801 2999682232 199999
> 3 0 318280 50602
> Jan 16 06:38:05 pqmon NOTE: 137978 1 62021 2999903568 199999
> 3 0 96944 50902
> Jan 16 06:43:05 pqmon NOTE: 125315 1 74684 2999967744 199999
> 3 0 32768 51202
> Jan 16 06:48:05 pqmon NOTE: 129348 2 70650 2999994696 199999
> 3 0 5600 51502
> Jan 16 06:53:05 pqmon NOTE: 123207 2 76791 2999843208 199999
> 3 0 150032 51802
> Jan 16 06:58:05 pqmon NOTE: 119818 1 80181 2999724576 199999
> 3 0 275936 52102
> Jan 16 07:03:05 pqmon NOTE: 125066 2 74932 2999869368 199999
> 3 0 129800 52402
> Jan 16 07:08:05 pqmon NOTE: 127885 1 72114 2999963784 199999
> 3 0 36728 52702
> Jan 16 07:13:05 pqmon NOTE: 133766 1 66233 2999942872 199999
> 3 0 57640 53002
> Jan 16 07:18:05 pqmon NOTE: 134672 1 65327 2997088792 199999
> 3 0 2911720 53302
> Jan 16 07:23:05 pqmon NOTE: 125830 1 74169 2999968240 199999
> 3 0 32272 53602
> Jan 16 07:28:05 pqmon NOTE: 113597 1 86402 2999864520 199999
> 3 0 135992 53902
> Jan 16 07:33:05 pqmon NOTE: 111186 1 88813 2999129304 199999
> 3 0 871208 54202
> Jan 16 07:38:05 pqmon NOTE: 120260 1 79739 2999356984 199999
> 3 0 643528 54502
> Jan 16 07:43:05 pqmon NOTE: 120766 1 79233 2999997832 199999
> 3 0 2680 54802
> Jan 16 07:48:05 pqmon NOTE: 122401 1 77598 2999812760 199999
> 3 0 187752 55102
> Jan 16 07:53:05 pqmon NOTE: 127216 1 72783 2999951208 199999
> 3 0 49304 55402
> Jan 16 07:58:05 pqmon NOTE: 128692 2 71306 2999949696 199999
> 3 0 50624 55702
> Jan 16 08:03:05 pqmon NOTE: 128125 1 71874 2999702336 199999
> 3 0 298176 56002
> Jan 16 08:08:05 pqmon NOTE: 126750 1 73249 2999812624 199999
> 3 0 187888 56302
> Jan 16 08:13:05 pqmon NOTE: 124456 1 75543 2999095896 199999
> 3 0 904616 56602
> Jan 16 08:18:05 pqmon NOTE: 127322 2 72676 2999958512 199999
> 3 0 41768 56902
> Jan 16 08:23:05 pqmon NOTE: 130802 2 69196 2999904864 199999
> 3 0 95456 57202
> Jan 16 08:28:05 pqmon NOTE: 126254 1 73745 2991866744 199999
> 3 0 8133768 57502
> Jan 16 08:33:05 pqmon NOTE: 126780 1 73219 2999967056 199999
> 3 0 33456 57802
> Jan 16 08:38:05 pqmon NOTE: 122933 1 77066 2999999848 199999
> 3 0 664 58102
> Jan 16 08:43:05 pqmon NOTE: 114002 1 85997 2999995048 199999
> 3 0 5464 58402
> Jan 16 08:48:05 pqmon NOTE: 113211 1 86788 2999939960 199999
> 3 0 60552 58702
> Jan 16 08:53:05 pqmon NOTE: 126829 1 73170 2998945200 199999
> 3 0 1055312 59002
> Jan 16 08:58:05 pqmon NOTE: 128754 1 71245 2999435616 199999
> 3 0 564896 59302
> Jan 16 09:03:05 pqmon NOTE: 118879 2 81119 2999989128 199999
> 3 0 7424 59602
> Jan 16 09:08:05 pqmon NOTE: 120351 1 79648 2999858776 199999
> 3 0 141736 59902
> Jan 16 09:13:05 pqmon NOTE: 125569 1 74430 2999988480 199999
> 3 0 12032 60202
> Jan 16 09:18:05 pqmon NOTE: 130555 1 69444 2999957688 199999
> 3 0 42824 60502
> Jan 16 09:23:05 pqmon NOTE: 123200 2 76798 2999945784 199999
> 3 0 52328 60802
> Jan 16 09:28:05 pqmon NOTE: 124044 1 75955 2999885408 199999
> 3 0 115104 61102
> Jan 16 09:33:05 pqmon NOTE: 124185 2 75813 2999923656 199999
> 3 0 75488 61402
> Jan 16 09:38:05 pqmon NOTE: 121014 1 78985 2999994200 199999
> 3 0 6312 61702
> Jan 16 09:43:05 pqmon NOTE: 113833 1 86166 2999968448 199999
> 3 0 32064 62002
> Jan 16 09:48:05 pqmon NOTE: 113824 1 86175 2999877176 199999
> 3 0 123336 62302
> Jan 16 09:53:05 pqmon NOTE: 117918 1 82081 2999496208 199999
> 3 0 504304 62602
> Jan 16 09:58:05 pqmon NOTE: 112297 1 87702 2999993344 199999
> 3 0 7168 62902
> Jan 16 10:03:05 pqmon NOTE: 112484 1 87515 2999956608 199999
> 3 0 43904 63202
> Jan 16 10:08:05 pqmon NOTE: 114048 2 85950 2999867840 199999
> 3 0 126808 63502
> Jan 16 10:13:05 pqmon NOTE: 113839 2 86159 2999988592 199999
> 3 0 11680 63802
> Jan 16 10:18:05 pqmon NOTE: 115940 1 84059 2999974600 199999
> 3 0 25912 64102
> Jan 16 10:23:05 pqmon NOTE: 118960 1 81039 2999696496 199999
> 3 0 304016 64402
> Jan 16 10:28:05 pqmon NOTE: 116236 2 83762 2999924952 199999
> 3 0 70216 64702
> Jan 16 10:33:05 pqmon NOTE: 124004 2 75994 2997040792 199999
> 3 0 2951680 65002
> Jan 16 10:38:05 pqmon NOTE: 126621 1 73378 2999983456 199999
> 3 0 17056 65302
> Jan 16 10:43:05 pqmon NOTE: 125444 1 74555 2999989752 199999
> 3 0 10760 65602
> Jan 16 10:48:05 pqmon NOTE: 121328 1 78671 2999953856 199999
> 3 0 46656 65902
> Jan 16 10:53:05 pqmon NOTE: 119675 2 80323 2998886320 199999
> 3 0 1084600 66202
> Jan 16 10:58:05 pqmon NOTE: 117090 1 82909 2999998176 199999
> 3 0 2336 66502
> Jan 16 11:03:05 pqmon NOTE: 108592 2 91406 2999023600 199999
> 3 0 646872 66802
> Jan 16 11:08:05 pqmon NOTE: 110615 2 89383 2999951776 199999
> 3 0 43720 67102
> Jan 16 11:13:05 pqmon NOTE: 109738 1 90261 2999083032 199999
> 3 0 917480 67402
> Jan 16 11:18:05 pqmon NOTE: 104722 1 95277 2999991648 199999
> 3 0 8864 67702
> Jan 16 11:23:05 pqmon NOTE: 95543 1 104456 2990078032 199999
> 3 0 9922480 68002
> Jan 16 11:28:05 pqmon NOTE: 101960 1 98039 2999493368 199999
> 3 0 507144 68302
> Jan 16 11:33:05 pqmon NOTE: 102930 2 97068 2999302816 199999
> 3 0 440632 68602
> Jan 16 11:38:05 pqmon NOTE: 103688 1 96311 2999486352 199999
> 3 0 514160 68902
> Jan 16 11:43:05 pqmon NOTE: 104502 1 95497 2999266448 199999
> 3 0 734064 69202
> Jan 16 11:48:05 pqmon NOTE: 103715 1 96284 2999853168 199999
> 3 0 147344 69502
> Jan 16 11:53:05 pqmon NOTE: 109279 1 90720 2997999016 199999
> 3 0 2001496 69802
> Jan 16 11:58:05 pqmon NOTE: 104288 1 95711 2999995952 199999
> 3 0 4560 70102
> Jan 16 12:03:05 pqmon NOTE: 103716 2 96282 2999942152 199999
> 3 0 57992 70402
> Jan 16 12:08:05 pqmon NOTE: 102451 1 97548 2999876408 199999
> 3 0 124104 70702
> Jan 16 12:13:05 pqmon NOTE: 105721 1 94278 2999718104 199999
> 3 0 282408 71002
> Jan 16 12:18:05 pqmon NOTE: 102483 1 97516 2983254456 199999
> 3 0 16746056 71302
> Jan 16 12:23:05 pqmon NOTE: 110753 1 89246 2999470920 199999
> 3 0 529592 71602
> Jan 16 12:28:05 pqmon NOTE: 112588 1 87411 2999854912 199999
> 3 0 145600 71902
> Jan 16 12:33:05 pqmon NOTE: 111663 2 88335 2998686896 199999
> 3 0 1114608 72202
> Jan 16 12:38:05 pqmon NOTE: 105013 1 94986 2999977664 199999
> 3 0 22848 72502
> Jan 16 12:43:05 pqmon NOTE: 109028 1 90971 2992194576 199999
> 3 0 7805936 72802
> Jan 16 12:48:05 pqmon NOTE: 108982 1 91017 3000000296 199999
> 3 0 216 73102
> Jan 16 12:53:05 pqmon NOTE: 110827 1 89172 2999880008 199999
> 3 0 120504 73402
> Jan 16 12:58:05 pqmon NOTE: 99670 1 100329 2999526936 199999
> 3 0 473576 73702
> Jan 16 13:03:05 pqmon NOTE: 102935 1 97064 2999980528 199999
> 3 0 19984 74002
> Jan 16 13:08:05 pqmon NOTE: 105077 0 94923 3000000512 199999
> 3 0 0 74302
> Jan 16 13:13:05 pqmon NOTE: 99958 2 100040 2999449840 199999
> 3 0 313192 74602
> Jan 16 13:18:05 pqmon NOTE: 103311 1 96688 2995839624 199999
> 3 0 4160888 74902
> Jan 16 13:23:05 pqmon NOTE: 108959 2 91039 2999982584 199999
> 3 0 17768 75202
> Jan 16 13:28:05 pqmon NOTE: 112787 1 87212 2998769720 199999
> 3 0 1230792 75502
> Jan 16 13:33:05 pqmon NOTE: 115691 2 84307 2999848216 199999
> 3 0 147784 75802
> Jan 16 13:38:05 pqmon NOTE: 120612 1 79387 2999625864 199999
> 3 0 374648 76102
> Jan 16 13:43:05 pqmon NOTE: 115633 2 84365 2999669816 199999
> 3 0 323160 76402
> Jan 16 13:48:05 pqmon NOTE: 119396 2 80602 2999536544 199999
> 3 0 463552 76702
> Jan 16 13:53:05 pqmon NOTE: 126014 2 73984 2999581264 199999
> 3 0 419008 77002
> Jan 16 13:58:05 pqmon NOTE: 117288 2 82710 2999921400 199999
> 3 0 76456 77302
> Jan 16 14:03:05 pqmon NOTE: 120417 2 79581 2998780160 199999
> 3 0 968264 77602
> Jan 16 14:08:05 pqmon NOTE: 121814 1 78185 2999724368 199999
> 3 0 276144 77902
> Jan 16 14:13:05 pqmon NOTE: 114444 2 85554 2999958744 199999
> 3 0 41336 78202
> Jan 16 14:18:05 pqmon NOTE: 100704 1 99295 2999990792 199999
> 3 0 9720 78502
> Jan 16 14:23:05 pqmon NOTE: 99111 2 100887 2999486480 199999
> 3 0 513456 78802
> Jan 16 14:28:05 pqmon NOTE: 90364 1 109635 2998573848 199999
> 3 0 1426664 79102
> Jan 16 14:33:05 pqmon NOTE: 93249 1 106750 2999982296 199999
> 3 0 18216 79402
> Jan 16 14:38:05 pqmon NOTE: 93826 2 106172 2997471920 199999
> 3 0 2143856 79702
> Jan 16 14:43:05 pqmon NOTE: 94575 1 105424 2999616000 199999
> 3 0 384512 80002
> Jan 16 14:48:05 pqmon NOTE: 93327 2 106671 2975330760 199999
> 3 0 24156600 80302
> Jan 16 14:53:05 pqmon NOTE: 110816 1 89183 2988839504 199999
> 3 0 11161008 80602
> Jan 16 14:58:05 pqmon NOTE: 111887 2 88111 2999588728 199999
> 3 0 370608 80902
> Jan 16 15:03:05 pqmon NOTE: 113089 3 86908 2998917248 199999
> 3 0 834984 81202
> Jan 16 15:08:05 pqmon NOTE: 114215 1 85784 2981162640 199999
> 3 0 18837872 81502
> Jan 16 15:13:05 pqmon NOTE: 117087 1 82912 2999889272 199999
> 3 0 111240 81802
> Jan 16 15:18:05 pqmon NOTE: 120026 0 79974 3000000512 199999
> 3 0 0 82102
> Jan 16 15:23:05 pqmon NOTE: 122294 2 77704 2999957328 199999
> 3 0 33592 82402
> Jan 16 15:28:05 pqmon NOTE: 126456 3 73541 2999954968 199999
> 3 0 36280 82702
> Jan 16 15:33:05 pqmon NOTE: 128278 1 71721 2999999696 199999
> 3 0 816 83002
> Jan 16 15:38:05 pqmon NOTE: 134706 2 65292 2999999344 199999
> 3 0 848 83302
> Jan 16 15:40:43 pqmon NOTE: Exiting
Regards,
Steve Emmerson
Ticket Details
===================
Ticket ID: GXL-914371
Department: Support LDM
Priority: Normal
Status: Closed
