[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #GXL-914371]: Strange LDM queue behavior - 6.11.6



Ryan,

The LDM product-queue is designed to delete products from the product-queue in 
order to make room for new products. Eligible products are removed in FIFO 
order. Products that are being processed (by pqact(1), for example) are not 
elegible are their deletion is skipped.

At about 2014-01-15T17:28:04, the age of the oldest product in the queue starts 
increasing at the rate of one second per second, while the number of products 
in the queue continues to fluctuate. I suspect that the oldest product in the 
queue is stuck because it's being processed by a pqact(1) process that is 
waiting on a decoder to finish reading a product from standard input.

Look for a hung pqact(1) process (one that's not accumulating any CPU time). 
Figure-out what decoder it's piping the product to and fix the decoder.

An alternative explanation is that a pqact(1) FILE action (rather than a PIPE 
action) is hung. This could happen if, for example, a product was being written 
to an NFS-mounted disk and the disk was somehow unavailable. This possibility 
is less likely in my opinion.

> We're running LDM 6.11.6 on RHEL 5.10.  Usually our queue retention time is
> around 2000-3000 for this system.  I noticed this morning that our product
> age is now about 83000.   Any ideas what would cause this?  I assume
> there's a product that's not being scoured out of the queue or something.
> I've been running a pqmon every 5 min for a few days, the output is below.
> 
> Jan 15 09:08:04 pqmon NOTE:  68880     2  131118  2999122344    199999
> 3         0    861416 1223
> Jan 15 09:13:04 pqmon NOTE:  55024     1  144975  2999994920    199999
> 3         0      5592 1180
> Jan 15 09:18:04 pqmon NOTE:  54444     1  145555  2999780712    199999
> 3         0    219800 1378
> Jan 15 09:23:04 pqmon NOTE:  61308     3  138689  2998617112    199999
> 3         0   1381056 1567
> Jan 15 09:28:04 pqmon NOTE:  68894     1  131105  2998004112    199999
> 3         0   1996400 1783
> Jan 15 09:33:04 pqmon NOTE:  77441     2  122557  2999720288    199999
> 3         0    279816 1984
> Jan 15 09:38:04 pqmon NOTE:  84783     1  115216  2999973840    199999
> 3         0     26672 2154
> Jan 15 09:43:04 pqmon NOTE:  90265     2  109733  2999056584    199999
> 3         0    819888 2311
> Jan 15 09:48:04 pqmon NOTE:  98532     1  101467  2997856288    199999
> 3         0   2144224 2555
> Jan 15 09:53:04 pqmon NOTE: 106470     2   93528  2999674784    199999
> 3         0    315192 2750
> Jan 15 09:58:04 pqmon NOTE: 115677     1   84322  2999989072    199999
> 3         0     11440 2959
> Jan 15 10:03:04 pqmon NOTE: 119168     1   80831  2999207560    199999
> 3         0    792952 3036
> Jan 15 10:08:04 pqmon NOTE: 122021     1   77978  2999993432    199999
> 3         0      7080 3060
> Jan 15 10:13:04 pqmon NOTE: 121645     2   78353  2999913648    199999
> 3         0     86560 3031
> Jan 15 10:18:04 pqmon NOTE: 128255     1   71744  2999989448    199999
> 3         0     11064 3032
> Jan 15 10:23:04 pqmon NOTE: 128989     1   71010  2994736184    199999
> 3         0   5264328 3073
> Jan 15 10:28:04 pqmon NOTE: 126473     1   73526  2999987928    199999
> 3         0     12584 3068
> Jan 15 10:33:04 pqmon NOTE: 124818     1   75181  2999545496    199999
> 3         0    455016 3045
> Jan 15 10:38:04 pqmon NOTE: 132622     1   67377  2999710512    199999
> 3         0    290000 3072
> Jan 15 10:43:04 pqmon NOTE: 127110     2   72888  2999746560    199999
> 3         0    250392 2961
> Jan 15 10:48:04 pqmon NOTE: 130480     1   69519  2999924272    199999
> 3         0     76240 3108
> Jan 15 10:53:04 pqmon NOTE: 135467     2   64531  2999750264    199999
> 3         0    250040 3302
> Jan 15 10:58:04 pqmon NOTE: 135531     2   64467  2999192032    199999
> 3         0    783136 3312
> Jan 15 11:03:04 pqmon NOTE: 129828     2   70170  2999942144    199999
> 3         0     54440 3196
> Jan 15 11:08:04 pqmon NOTE: 118541     1   81458  2996797888    199999
> 3         0   3202624 3085
> Jan 15 11:13:04 pqmon NOTE: 114348     1   85651  2999986440    199999
> 3         0     14072 2996
> Jan 15 11:18:04 pqmon NOTE: 112982     1   87017  2999970768    199999
> 3         0     29744 2943
> Jan 15 11:23:04 pqmon NOTE: 105589     1   94410  2999992344    199999
> 3         0      8168 2833
> Jan 15 11:28:04 pqmon NOTE: 105142     1   94857  2999112248    199999
> 3         0    888264 2851
> Jan 15 11:33:04 pqmon NOTE: 110261     1   89738  2999520792    199999
> 3         0    479720 3017
> Jan 15 11:38:04 pqmon NOTE:  99409     2  100589  2999650288    199999
> 3         0    327272 2664
> Jan 15 11:43:04 pqmon NOTE: 101993     1   98006  2999994696    199999
> 3         0      5816 2756
> Jan 15 11:48:04 pqmon NOTE:  98880     1  101119  2994070432    199999
> 3         0   5930080 2737
> Jan 15 11:53:04 pqmon NOTE: 102255     2   97743  2999998792    199999
> 3         0      1328 2863
> Jan 15 11:58:04 pqmon NOTE: 106212     1   93787  2999560560    199999
> 3         0    439952 2957
> Jan 15 12:03:04 pqmon NOTE: 113163     2   86835  2999649544    199999
> 3         0    350416 3144
> Jan 15 12:08:04 pqmon NOTE: 111220     2   88778  2999996288    199999
> 3         0      3320 3042
> Jan 15 12:13:04 pqmon NOTE: 112620     1   87379  2999932536    199999
> 3         0     67976 3071
> Jan 15 12:18:04 pqmon NOTE: 116997     1   83002  2999994240    199999
> 3         0      6272 3201
> Jan 15 12:23:04 pqmon NOTE: 119552     1   80447  2999975224    199999
> 3         0     25288 3252
> Jan 15 12:28:04 pqmon NOTE: 120367     1   79632  2999869752    199999
> 3         0    130760 3251
> Jan 15 12:33:04 pqmon NOTE: 122922     1   77077  2999739160    199999
> 3         0    261352 3330
> Jan 15 12:38:04 pqmon NOTE: 127240     2   72758  2999438992    199999
> 3         0    557880 3423
> Jan 15 12:43:04 pqmon NOTE: 125748     2   74250  2999233376    199999
> 3         0    764792 3384
> Jan 15 12:48:04 pqmon NOTE: 122368     2   77630  2999880184    199999
> 3         0    119104 3262
> Jan 15 12:53:04 pqmon NOTE: 121420     1   78579  2999921296    199999
> 3         0     79216 3278
> Jan 15 12:58:04 pqmon NOTE: 113932     1   86067  2999232656    199999
> 3         0    767856 3099
> Jan 15 13:03:04 pqmon NOTE: 110314     2   89684  2999923560    199999
> 3         0     76584 3048
> Jan 15 13:08:04 pqmon NOTE: 110522     2   89476  2998873440    199999
> 3         0   1126776 3054
> Jan 15 13:13:04 pqmon NOTE: 108993     2   91005  2999755224    199999
> 3         0    229864 3035
> Jan 15 13:18:04 pqmon NOTE: 106514     1   93485  2999996656    199999
> 3         0      3856 2978
> Jan 15 13:23:04 pqmon NOTE: 104838     2   95160  2999997288    199999
> 3         0      1728 2918
> Jan 15 13:28:04 pqmon NOTE: 106363     1   93636  2999998832    199999
> 3         0      1680 2891
> Jan 15 13:33:04 pqmon NOTE: 104246     1   95753  2997189176    199999
> 3         0   2811336 2765
> Jan 15 13:38:04 pqmon NOTE: 105130     1   94869  2990797880    199999
> 3         0   9202632 2732
> Jan 15 13:43:04 pqmon NOTE: 110527     2   89471  2999346392    199999
> 3         0    651960 2851
> Jan 15 13:48:04 pqmon NOTE: 111165     1   88834  2999791976    199999
> 3         0    208536 2833
> Jan 15 13:53:04 pqmon NOTE: 117811     2   82187  2999355152    199999
> 3         0    644040 3008
> Jan 15 13:58:04 pqmon NOTE: 113827     2   86171  2997953408    199999
> 3         0   2044088 2852
> Jan 15 14:03:04 pqmon NOTE: 115986     2   84012  2999998880    199999
> 3         0      1304 2863
> Jan 15 14:08:04 pqmon NOTE: 123373     1   76626  3000000160    199999
> 3         0       352 3044
> Jan 15 14:13:04 pqmon NOTE: 122248     1   77751  2999920640    199999
> 3         0     79872 2994
> Jan 15 14:18:04 pqmon NOTE: 129551     1   70448  2999895544    199999
> 3         0    104968 3194
> Jan 15 14:23:04 pqmon NOTE: 122187     2   77811  2999575472    199999
> 3         0    357024 3025
> Jan 15 14:28:04 pqmon NOTE: 110758     1   89241  2999944960    199999
> 3         0     55552 2754
> Jan 15 14:33:04 pqmon NOTE: 110561     2   89437  2998889440    199999
> 3         0   1109416 2746
> Jan 15 14:38:04 pqmon NOTE: 104651     2   95347  2999178096    199999
> 3         0    520832 2630
> Jan 15 14:43:04 pqmon NOTE: 108676     2   91322  2999623920    199999
> 3         0    346408 2799
> Jan 15 14:48:04 pqmon NOTE: 101841     1   98158  3000000096    199999
> 3         0       416 2676
> Jan 15 14:53:04 pqmon NOTE: 114836     2   85162  2998919640    199999
> 3         0    705176 2516
> Jan 15 14:58:04 pqmon NOTE: 106137     1   93862  2999344392    199999
> 3         0    656120 2280
> Jan 15 15:03:04 pqmon NOTE: 110189     2   89809  2999970552    199999
> 3         0     22632 2413
> Jan 15 15:08:04 pqmon NOTE: 115260     1   84739  2999329032    199999
> 3         0    671480 2565
> Jan 15 15:13:04 pqmon NOTE: 117627     2   82371  2987104840    199999
> 3         0  12243816 2641
> Jan 15 15:18:04 pqmon NOTE: 119851     1   80148  2997499632    199999
> 3         0   2500880 2712
> Jan 15 15:23:04 pqmon NOTE: 119855     2   80143  2999875000    199999
> 3         0     79992 2681
> Jan 15 15:28:04 pqmon NOTE: 122140     2   77858  2999780784    199999
> 3         0    219048 2690
> Jan 15 15:33:04 pqmon NOTE: 127669     1   72330  2999304608    199999
> 3         0    695904 2795
> Jan 15 15:38:04 pqmon NOTE: 136797     2   63201  2999991384    199999
> 3         0      8896 3039
> Jan 15 15:43:04 pqmon NOTE: 134651     1   65348  2999997992    199999
> 3         0      2520 3164
> Jan 15 15:48:04 pqmon NOTE: 128520     1   71479  2999996560    199999
> 3         0      3952 3314
> Jan 15 15:53:04 pqmon NOTE: 136632     1   63367  2999940688    199999
> 3         0     59824 3548
> Jan 15 15:58:04 pqmon NOTE: 140775     1   59224  2999788216    199999
> 3         0    212296 3622
> Jan 15 16:03:04 pqmon NOTE: 140951     1   59048  2999081632    199999
> 3         0    918880 3613
> Jan 15 16:08:04 pqmon NOTE: 152188     1   47811  2999135712    199999
> 3         0    864800 3757
> Jan 15 16:13:04 pqmon NOTE: 158953     2   41045  2999933752    199999
> 3         0     66472 3892
> Jan 15 16:18:04 pqmon NOTE: 161553     1   38446  2987672208    199999
> 3         0  12328304 3957
> Jan 15 16:23:04 pqmon NOTE: 160932     1   39067  2999996392    199999
> 3         0      4120 3951
> Jan 15 16:28:04 pqmon NOTE: 163008     1   36991  2999943072    199999
> 3         0     57440 3870
> Jan 15 16:33:04 pqmon NOTE: 161850     2   38148  2999966600    199999
> 3         0     33720 3876
> Jan 15 16:38:04 pqmon NOTE: 166190     1   33809  2999299168    199999
> 3         0    701344 3996
> Jan 15 16:43:04 pqmon NOTE: 162019     1   37980  2999950352    199999
> 3         0     50160 3947
> Jan 15 16:48:04 pqmon NOTE: 160208     1   39791  2999280456    199999
> 3         0    720056 4034
> Jan 15 16:53:04 pqmon NOTE: 165492     1   34507  2999942528    199999
> 3         0     57984 4202
> Jan 15 16:58:04 pqmon NOTE: 157311     2   42687  2999941136    199999
> 3         0     33560 3934
> Jan 15 17:03:04 pqmon NOTE: 158121     1   41878  2991983080    199999
> 3         0   8017432 3985
> Jan 15 17:08:04 pqmon NOTE: 161494     1   38505  2999981824    199999
> 3         0     18688 4109
> Jan 15 17:13:04 pqmon NOTE: 164410     1   35589  2996538456    199999
> 3         0   3462056 4212
> Jan 15 17:18:04 pqmon NOTE: 162553     1   37446  2999996232    199999
> 3         0      4280 4277
> Jan 15 17:23:04 pqmon NOTE: 153369     1   46630  2999987688    199999
> 3         0     12824 4135
> Jan 15 17:28:04 pqmon NOTE: 149446     1   50553  2999815304    199999
> 3         0    185208 4022
> Jan 15 17:33:04 pqmon NOTE: 150864     2   49134  2990187640    199999
> 3         0   9812464 4110
> Jan 15 17:38:04 pqmon NOTE: 141671     1   58328  2994037944    199999
> 3         0   5962568 4102
> Jan 15 17:43:04 pqmon NOTE: 132960     1   67039  2992575384    199999
> 3         0   7425128 4402
> Jan 15 17:48:04 pqmon NOTE: 130732     1   69267  2999442320    199999
> 3         0    558192 4702
> Jan 15 17:53:04 pqmon NOTE: 128911     2   71087  2999426528    199999
> 3         0    543232 5002
> Jan 15 17:58:04 pqmon NOTE: 128692     2   71306  2999973944    199999
> 3         0     23728 5302
> Jan 15 18:03:04 pqmon NOTE: 117698     1   82301  2999886408    199999
> 3         0    114104 5602
> Jan 15 18:08:04 pqmon NOTE: 115135     1   84864  2999935096    199999
> 3         0     65416 5902
> Jan 15 18:13:04 pqmon NOTE: 114673     2   85325  2999929440    199999
> 3         0     69128 6202
> Jan 15 18:18:04 pqmon NOTE: 119792     1   80207  2999438136    199999
> 3         0    562376 6502
> Jan 15 18:23:04 pqmon NOTE: 123103     2   76895  2999854392    199999
> 3         0    102696 6802
> Jan 15 18:28:04 pqmon NOTE: 121993     1   78006  2999077648    199999
> 3         0    922864 7102
> Jan 15 18:33:04 pqmon NOTE: 122147     2   77851  2999684704    199999
> 3         0    305064 7402
> Jan 15 18:38:04 pqmon NOTE: 116716     2   83282  2995443440    199999
> 3         0   4556600 7702
> Jan 15 18:43:04 pqmon NOTE: 103193     1   96806  2999999344    199999
> 3         0      1168 8002
> Jan 15 18:48:04 pqmon NOTE: 104729     1   95270  2999335208    199999
> 3         0    665304 8302
> Jan 15 18:53:04 pqmon NOTE: 111790     3   88207  2998877768    199999
> 3         0    806472 8602
> Jan 15 18:58:04 pqmon NOTE: 108890     1   91109  2986636600    199999
> 3         0  13363912 8902
> Jan 15 19:03:04 pqmon NOTE: 113426     2   86572  2999985008    199999
> 3         0     14656 9202
> Jan 15 19:08:04 pqmon NOTE: 104617     1   95382  2999997920    199999
> 3         0      2592 9502
> Jan 15 19:13:04 pqmon NOTE: 104698     2   95300  2999836568    199999
> 3         0    115760 9802
> Jan 15 19:18:04 pqmon NOTE: 104097     1   95902  2999998744    199999
> 3         0      1768 10102
> Jan 15 19:23:04 pqmon NOTE: 103259     2   96739  2999590472    199999
> 3         0    404680 10402
> Jan 15 19:28:04 pqmon NOTE: 101465     2   98533  2999995680    199999
> 3         0      2928 10702
> Jan 15 19:33:04 pqmon NOTE: 105548     1   94451  2999773320    199999
> 3         0    227192 11002
> Jan 15 19:38:04 pqmon NOTE: 106022     1   93977  2999193360    199999
> 3         0    807152 11302
> Jan 15 19:43:04 pqmon NOTE:  98425     1  101574  2995830112    199999
> 3         0   4170400 11602
> Jan 15 19:48:04 pqmon NOTE:  93846     1  106153  2999412800    199999
> 3         0    587712 11902
> Jan 15 19:53:04 pqmon NOTE:  97186     1  102813  2999989928    199999
> 3         0     10584 12202
> Jan 15 19:58:04 pqmon NOTE:  93350     2  106648  2998100720    199999
> 3         0   1651104 12502
> Jan 15 20:03:04 pqmon NOTE:  92681     1  107318  2999996496    199999
> 3         0      4016 12802
> Jan 15 20:08:04 pqmon NOTE:  96325     2  103673  2999995840    199999
> 3         0      2336 13102
> Jan 15 20:13:04 pqmon NOTE:  90580     1  109419  2999992160    199999
> 3         0      8352 13402
> Jan 15 20:18:04 pqmon NOTE:  91078     1  108921  2999982224    199999
> 3         0     18288 13702
> Jan 15 20:23:04 pqmon NOTE:  97475     1  102524  2998476896    199999
> 3         0   1523616 14002
> Jan 15 20:28:04 pqmon NOTE:  98080     2  101918  2999541280    199999
> 3         0    422984 14302
> Jan 15 20:33:04 pqmon NOTE: 100597     1   99402  2999067240    199999
> 3         0    933272 14602
> Jan 15 20:38:04 pqmon NOTE: 108308     1   91691  2983436232    199999
> 3         0  16564280 14902
> Jan 15 20:43:04 pqmon NOTE: 107329     1   92670  2999100976    199999
> 3         0    899536 15202
> Jan 15 20:48:04 pqmon NOTE: 115074     1   84925  2996470824    199999
> 3         0   3529688 15502
> Jan 15 20:53:04 pqmon NOTE: 132062     1   67937  2999999112    199999
> 3         0      1400 15802
> Jan 15 20:58:04 pqmon NOTE: 128502     1   71497  2999390240    199999
> 3         0    610272 16102
> Jan 15 21:03:04 pqmon NOTE: 131351     1   68648  2988527008    199999
> 3         0  11473504 16402
> Jan 15 21:08:04 pqmon NOTE: 129926     1   70073  2998905968    199999
> 3         0   1094544 16702
> Jan 15 21:13:04 pqmon NOTE: 130613     2   69385  2999992216    199999
> 3         0      6616 17002
> Jan 15 21:18:04 pqmon NOTE: 133114     1   66885  2989062680    199999
> 3         0  10937832 17302
> Jan 15 21:23:04 pqmon NOTE: 129279     1   70720  2988288368    199999
> 3         0  11712144 17602
> Jan 15 21:28:04 pqmon NOTE: 122838     2   77160  2999707552    199999
> 3         0    286264 17902
> Jan 15 21:33:04 pqmon NOTE: 110772     1   89227  2999995416    199999
> 3         0      5096 18202
> Jan 15 21:38:04 pqmon NOTE: 106218     1   93781  2999996160    199999
> 3         0      4352 18502
> Jan 15 21:43:04 pqmon NOTE: 106253     1   93746  2999115000    199999
> 3         0    885512 18802
> Jan 15 21:48:04 pqmon NOTE: 104549     1   95450  2999811344    199999
> 3         0    189168 19102
> Jan 15 21:53:04 pqmon NOTE: 106494     1   93505  2999476176    199999
> 3         0    524336 19402
> Jan 15 21:58:04 pqmon NOTE: 100023     2   99975  2999845032    199999
> 3         0    122728 19702
> Jan 15 22:03:04 pqmon NOTE: 103816     3   96181  2999954216    199999
> 3         0     32928 20002
> Jan 15 22:08:04 pqmon NOTE: 109079     3   90918  2999930760    199999
> 3         0     62776 20302
> Jan 15 22:13:05 pqmon NOTE: 114698     1   85301  2993996000    199999
> 3         0   6004512 20602
> Jan 15 22:18:05 pqmon NOTE: 104681     1   95318  2999978520    199999
> 3         0     21992 20902
> Jan 15 22:23:05 pqmon NOTE: 105218     1   94781  2985611776    199999
> 3         0  14388736 21202
> Jan 15 22:28:05 pqmon NOTE: 117269     2   82729  2998557480    199999
> 3         0   1426792 21502
> Jan 15 22:33:05 pqmon NOTE: 120017     1   79982  2999991200    199999
> 3         0      9312 21802
> Jan 15 22:38:05 pqmon NOTE: 129011     1   70988  2999677392    199999
> 3         0    323120 22102
> Jan 15 22:43:05 pqmon NOTE: 130091     1   69908  2999985976    199999
> 3         0     14536 22402
> Jan 15 22:48:05 pqmon NOTE: 132076     1   67923  2999663848    199999
> 3         0    336664 22702
> Jan 15 22:53:05 pqmon NOTE: 134653     1   65346  2999996240    199999
> 3         0      4272 23002
> Jan 15 22:58:05 pqmon NOTE: 130713     1   69286  2999340400    199999
> 3         0    660112 23302
> Jan 15 23:03:05 pqmon NOTE: 122173     1   77826  2999368120    199999
> 3         0    632392 23602
> Jan 15 23:08:05 pqmon NOTE: 117696     1   82303  2999925168    199999
> 3         0     75344 23902
> Jan 15 23:13:05 pqmon NOTE: 119009     1   80990  2999826272    199999
> 3         0    174240 24202
> Jan 15 23:18:05 pqmon NOTE: 122917     1   77082  2982466784    199999
> 3         0  17533728 24502
> Jan 15 23:23:05 pqmon NOTE: 100131     1   99868  2999988928    199999
> 3         0     11584 24802
> Jan 15 23:28:05 pqmon NOTE:  91664     1  108335  2999978368    199999
> 3         0     22144 25102
> Jan 15 23:33:05 pqmon NOTE:  89866     1  110133  2999277296    199999
> 3         0    723216 25402
> Jan 15 23:38:05 pqmon NOTE:  96038     1  103961  2982654512    199999
> 3         0  17346000 25702
> Jan 15 23:43:05 pqmon NOTE:  97076     2  102922  2999921360    199999
> 3         0     62768 26002
> Jan 15 23:48:05 pqmon NOTE:  99106     1  100893  2999988920    199999
> 3         0     11592 26302
> Jan 15 23:53:05 pqmon NOTE:  98860     2  101138  2999964976    199999
> 3         0     35344 26602
> Jan 15 23:58:05 pqmon NOTE:  89755     1  110244  2999880752    199999
> 3         0    119760 26902
> Jan 16 00:03:05 pqmon NOTE:  91765     1  108234  2992141912    199999
> 3         0   7858600 27202
> Jan 16 00:08:05 pqmon NOTE:  99651     2  100347  2999015632    199999
> 3         0    793232 27502
> Jan 16 00:13:05 pqmon NOTE: 107222     1   92777  2999807680    199999
> 3         0    192832 27802
> Jan 16 00:18:05 pqmon NOTE: 115100     1   84899  2987289648    199999
> 3         0  12710864 28102
> Jan 16 00:23:05 pqmon NOTE: 115670     2   84328  2999962496    199999
> 3         0     37800 28402
> Jan 16 00:28:05 pqmon NOTE: 110806     1   89193  2998697688    199999
> 3         0   1302824 28702
> Jan 16 00:33:05 pqmon NOTE: 116684     3   83313  2999004208    199999
> 3         0    903680 29002
> Jan 16 00:38:05 pqmon NOTE: 118185     1   81814  2997243824    199999
> 3         0   2756688 29302
> Jan 16 00:43:05 pqmon NOTE: 112997     2   87001  2999352864    199999
> 3         0    647456 29602
> Jan 16 00:48:05 pqmon NOTE: 113986     1   86013  2999298256    199999
> 3         0    702256 29902
> Jan 16 00:53:05 pqmon NOTE: 113483     1   86516  2998141440    199999
> 3         0   1859072 30202
> Jan 16 00:58:05 pqmon NOTE: 111311     1   88688  2999996280    199999
> 3         0      4232 30502
> Jan 16 01:03:05 pqmon NOTE: 109354     2   90644  2999980528    199999
> 3         0     18384 30802
> Jan 16 01:08:05 pqmon NOTE:  99753     2  100245  2999827456    199999
> 3         0    170344 31102
> Jan 16 01:13:05 pqmon NOTE:  97838     3  102159  2999759208    199999
> 3         0    105464 31402
> Jan 16 01:18:05 pqmon NOTE:  93491     2  106507  2999331112    199999
> 3         0    669016 31702
> Jan 16 01:23:05 pqmon NOTE:  95106     2  104892  2999079864    199999
> 3         0    885584 32002
> Jan 16 01:28:05 pqmon NOTE:  99128     2  100870  2999810472    199999
> 3         0    189504 32302
> Jan 16 01:33:05 pqmon NOTE:  98628     3  101369  2998785312    199999
> 3         0    974576 32602
> Jan 16 01:38:05 pqmon NOTE:  95143     1  104856  2999549824    199999
> 3         0    450688 32902
> Jan 16 01:43:05 pqmon NOTE:  93997     1  106002  2998081784    199999
> 3         0   1918728 33202
> Jan 16 01:48:05 pqmon NOTE: 100976     1   99023  2986613832    199999
> 3         0  13386680 33502
> Jan 16 01:53:05 pqmon NOTE: 105482     1   94517  2999830160    199999
> 3         0    170352 33802
> Jan 16 01:58:05 pqmon NOTE: 107343     2   92655  2999963024    199999
> 3         0     33816 34102
> Jan 16 02:03:05 pqmon NOTE: 110823     1   89176  2999800696    199999
> 3         0    199816 34402
> Jan 16 02:08:05 pqmon NOTE: 111959     1   88040  2999951936    199999
> 3         0     48576 34702
> Jan 16 02:13:05 pqmon NOTE: 107900     2   92098  2999911832    199999
> 3         0     73056 35002
> Jan 16 02:18:05 pqmon NOTE: 108927     1   91072  2998169936    199999
> 3         0   1830576 35302
> Jan 16 02:23:05 pqmon NOTE: 110916     1   89083  2999964528    199999
> 3         0     35984 35602
> Jan 16 02:28:05 pqmon NOTE: 110144     1   89855  2999990840    199999
> 3         0      9672 35902
> Jan 16 02:33:05 pqmon NOTE:  96957     2  103041  2999767064    199999
> 3         0    198792 36202
> Jan 16 02:38:05 pqmon NOTE: 102963     2   97035  2999986296    199999
> 3         0     13992 36502
> Jan 16 02:43:05 pqmon NOTE:  98896     2  101102  2999653344    199999
> 3         0    323848 36802
> Jan 16 02:48:05 pqmon NOTE:  94976     1  105023  2999416584    199999
> 3         0    583928 37102
> Jan 16 02:53:05 pqmon NOTE: 113625     2   86373  2999769864    199999
> 3         0    206232 37402
> Jan 16 02:58:05 pqmon NOTE: 113852     1   86147  2998730080    199999
> 3         0   1270432 37702
> Jan 16 03:03:05 pqmon NOTE: 115495     1   84504  2999990496    199999
> 3         0     10016 38002
> Jan 16 03:08:05 pqmon NOTE: 117448     1   82551  2985156312    199999
> 3         0  14844200 38302
> Jan 16 03:13:05 pqmon NOTE: 113020     1   86979  2997540336    199999
> 3         0   2460176 38602
> Jan 16 03:18:05 pqmon NOTE: 119349     2   80649  2999999528    199999
> 3         0       776 38902
> Jan 16 03:23:05 pqmon NOTE: 120601     2   79397  2999908712    199999
> 3         0     90840 39202
> Jan 16 03:28:05 pqmon NOTE: 125270     2   74728  2999082600    199999
> 3         0    917584 39502
> Jan 16 03:33:05 pqmon NOTE: 132394     1   67605  2999998584    199999
> 3         0      1928 39802
> Jan 16 03:38:05 pqmon NOTE: 141688     2   58310  2999992728    199999
> 3         0      6088 40102
> Jan 16 03:43:05 pqmon NOTE: 135480     1   64519  2999264864    199999
> 3         0    735648 40402
> Jan 16 03:48:05 pqmon NOTE: 136152     1   63847  2999998024    199999
> 3         0      2488 40702
> Jan 16 03:53:05 pqmon NOTE: 138600     2   61398  2999894280    199999
> 3         0    106040 41002
> Jan 16 03:58:05 pqmon NOTE: 142133     2   57865  2989110544    199999
> 3         0   8752000 41302
> Jan 16 04:03:05 pqmon NOTE: 143839     2   56159  2999900752    199999
> 3         0     97256 41602
> Jan 16 04:08:05 pqmon NOTE: 152826     2   47172  2999211496    199999
> 3         0    766528 41902
> Jan 16 04:13:05 pqmon NOTE: 160407     1   39592  2999991936    199999
> 3         0      8576 42202
> Jan 16 04:18:05 pqmon NOTE: 166602     2   33396  2998493472    199999
> 3         0   1504984 42502
> Jan 16 04:23:05 pqmon NOTE: 169154     1   30845  2999949696    199999
> 3         0     50816 42802
> Jan 16 04:28:05 pqmon NOTE: 171322     1   28677  2999995824    199999
> 3         0      4688 43102
> Jan 16 04:33:05 pqmon NOTE: 168886     2   31112  2999917064    199999
> 3         0     81904 43402
> Jan 16 04:38:05 pqmon NOTE: 172142     1   27857  2999983784    199999
> 3         0     16728 43702
> Jan 16 04:43:05 pqmon NOTE: 162336     2   37662  2999683904    199999
> 3         0    313600 44002
> Jan 16 04:48:05 pqmon NOTE: 162987     1   37012  2999947096    199999
> 3         0     53416 44302
> Jan 16 04:53:05 pqmon NOTE: 160293     1   39706  2999980416    199999
> 3         0     20096 44602
> Jan 16 04:58:05 pqmon NOTE: 154319     1   45680  2999994616    199999
> 3         0      5896 44902
> Jan 16 05:03:05 pqmon NOTE: 147510     1   52489  2998331104    199999
> 3         0   1669408 45202
> Jan 16 05:08:05 pqmon NOTE: 132937     2   67061  2999757280    199999
> 3         0    202664 45502
> Jan 16 05:13:05 pqmon NOTE: 129357     1   70642  2996257240    199999
> 3         0   3743272 45802
> Jan 16 05:18:05 pqmon NOTE: 122919     1   77080  2998928256    199999
> 3         0   1072256 46102
> Jan 16 05:23:05 pqmon NOTE: 116004     1   83995  2999992712    199999
> 3         0      7800 46402
> Jan 16 05:28:05 pqmon NOTE: 112696     1   87303  2999987416    199999
> 3         0     13096 46702
> Jan 16 05:33:05 pqmon NOTE: 112741     1   87258  2999352944    199999
> 3         0    647568 47002
> Jan 16 05:38:05 pqmon NOTE: 111395     1   88604  2999310992    199999
> 3         0    689520 47302
> Jan 16 05:43:05 pqmon NOTE: 110392     1   89607  2999492744    199999
> 3         0    507768 47602
> Jan 16 05:48:05 pqmon NOTE: 104553     1   95446  2999795472    199999
> 3         0    205040 47902
> Jan 16 05:53:05 pqmon NOTE: 115640     1   84359  2998223776    199999
> 3         0   1776736 48202
> Jan 16 05:58:05 pqmon NOTE: 124619     1   75380  2999976216    199999
> 3         0     24296 48502
> Jan 16 06:03:05 pqmon NOTE: 131891     1   68108  2998730856    199999
> 3         0   1269656 48802
> Jan 16 06:08:05 pqmon NOTE: 125340     1   74659  2999916480    199999
> 3         0     84032 49102
> Jan 16 06:13:05 pqmon NOTE: 115240     1   84759  2999048872    199999
> 3         0    951640 49402
> Jan 16 06:18:05 pqmon NOTE: 115878     1   84121  2999034040    199999
> 3         0    966472 49702
> Jan 16 06:23:05 pqmon NOTE: 120552     1   79447  2985583128    199999
> 3         0  14417384 50002
> Jan 16 06:28:05 pqmon NOTE: 125155     1   74844  2999199200    199999
> 3         0    801312 50302
> Jan 16 06:33:05 pqmon NOTE: 129198     1   70801  2999682232    199999
> 3         0    318280 50602
> Jan 16 06:38:05 pqmon NOTE: 137978     1   62021  2999903568    199999
> 3         0     96944 50902
> Jan 16 06:43:05 pqmon NOTE: 125315     1   74684  2999967744    199999
> 3         0     32768 51202
> Jan 16 06:48:05 pqmon NOTE: 129348     2   70650  2999994696    199999
> 3         0      5600 51502
> Jan 16 06:53:05 pqmon NOTE: 123207     2   76791  2999843208    199999
> 3         0    150032 51802
> Jan 16 06:58:05 pqmon NOTE: 119818     1   80181  2999724576    199999
> 3         0    275936 52102
> Jan 16 07:03:05 pqmon NOTE: 125066     2   74932  2999869368    199999
> 3         0    129800 52402
> Jan 16 07:08:05 pqmon NOTE: 127885     1   72114  2999963784    199999
> 3         0     36728 52702
> Jan 16 07:13:05 pqmon NOTE: 133766     1   66233  2999942872    199999
> 3         0     57640 53002
> Jan 16 07:18:05 pqmon NOTE: 134672     1   65327  2997088792    199999
> 3         0   2911720 53302
> Jan 16 07:23:05 pqmon NOTE: 125830     1   74169  2999968240    199999
> 3         0     32272 53602
> Jan 16 07:28:05 pqmon NOTE: 113597     1   86402  2999864520    199999
> 3         0    135992 53902
> Jan 16 07:33:05 pqmon NOTE: 111186     1   88813  2999129304    199999
> 3         0    871208 54202
> Jan 16 07:38:05 pqmon NOTE: 120260     1   79739  2999356984    199999
> 3         0    643528 54502
> Jan 16 07:43:05 pqmon NOTE: 120766     1   79233  2999997832    199999
> 3         0      2680 54802
> Jan 16 07:48:05 pqmon NOTE: 122401     1   77598  2999812760    199999
> 3         0    187752 55102
> Jan 16 07:53:05 pqmon NOTE: 127216     1   72783  2999951208    199999
> 3         0     49304 55402
> Jan 16 07:58:05 pqmon NOTE: 128692     2   71306  2999949696    199999
> 3         0     50624 55702
> Jan 16 08:03:05 pqmon NOTE: 128125     1   71874  2999702336    199999
> 3         0    298176 56002
> Jan 16 08:08:05 pqmon NOTE: 126750     1   73249  2999812624    199999
> 3         0    187888 56302
> Jan 16 08:13:05 pqmon NOTE: 124456     1   75543  2999095896    199999
> 3         0    904616 56602
> Jan 16 08:18:05 pqmon NOTE: 127322     2   72676  2999958512    199999
> 3         0     41768 56902
> Jan 16 08:23:05 pqmon NOTE: 130802     2   69196  2999904864    199999
> 3         0     95456 57202
> Jan 16 08:28:05 pqmon NOTE: 126254     1   73745  2991866744    199999
> 3         0   8133768 57502
> Jan 16 08:33:05 pqmon NOTE: 126780     1   73219  2999967056    199999
> 3         0     33456 57802
> Jan 16 08:38:05 pqmon NOTE: 122933     1   77066  2999999848    199999
> 3         0       664 58102
> Jan 16 08:43:05 pqmon NOTE: 114002     1   85997  2999995048    199999
> 3         0      5464 58402
> Jan 16 08:48:05 pqmon NOTE: 113211     1   86788  2999939960    199999
> 3         0     60552 58702
> Jan 16 08:53:05 pqmon NOTE: 126829     1   73170  2998945200    199999
> 3         0   1055312 59002
> Jan 16 08:58:05 pqmon NOTE: 128754     1   71245  2999435616    199999
> 3         0    564896 59302
> Jan 16 09:03:05 pqmon NOTE: 118879     2   81119  2999989128    199999
> 3         0      7424 59602
> Jan 16 09:08:05 pqmon NOTE: 120351     1   79648  2999858776    199999
> 3         0    141736 59902
> Jan 16 09:13:05 pqmon NOTE: 125569     1   74430  2999988480    199999
> 3         0     12032 60202
> Jan 16 09:18:05 pqmon NOTE: 130555     1   69444  2999957688    199999
> 3         0     42824 60502
> Jan 16 09:23:05 pqmon NOTE: 123200     2   76798  2999945784    199999
> 3         0     52328 60802
> Jan 16 09:28:05 pqmon NOTE: 124044     1   75955  2999885408    199999
> 3         0    115104 61102
> Jan 16 09:33:05 pqmon NOTE: 124185     2   75813  2999923656    199999
> 3         0     75488 61402
> Jan 16 09:38:05 pqmon NOTE: 121014     1   78985  2999994200    199999
> 3         0      6312 61702
> Jan 16 09:43:05 pqmon NOTE: 113833     1   86166  2999968448    199999
> 3         0     32064 62002
> Jan 16 09:48:05 pqmon NOTE: 113824     1   86175  2999877176    199999
> 3         0    123336 62302
> Jan 16 09:53:05 pqmon NOTE: 117918     1   82081  2999496208    199999
> 3         0    504304 62602
> Jan 16 09:58:05 pqmon NOTE: 112297     1   87702  2999993344    199999
> 3         0      7168 62902
> Jan 16 10:03:05 pqmon NOTE: 112484     1   87515  2999956608    199999
> 3         0     43904 63202
> Jan 16 10:08:05 pqmon NOTE: 114048     2   85950  2999867840    199999
> 3         0    126808 63502
> Jan 16 10:13:05 pqmon NOTE: 113839     2   86159  2999988592    199999
> 3         0     11680 63802
> Jan 16 10:18:05 pqmon NOTE: 115940     1   84059  2999974600    199999
> 3         0     25912 64102
> Jan 16 10:23:05 pqmon NOTE: 118960     1   81039  2999696496    199999
> 3         0    304016 64402
> Jan 16 10:28:05 pqmon NOTE: 116236     2   83762  2999924952    199999
> 3         0     70216 64702
> Jan 16 10:33:05 pqmon NOTE: 124004     2   75994  2997040792    199999
> 3         0   2951680 65002
> Jan 16 10:38:05 pqmon NOTE: 126621     1   73378  2999983456    199999
> 3         0     17056 65302
> Jan 16 10:43:05 pqmon NOTE: 125444     1   74555  2999989752    199999
> 3         0     10760 65602
> Jan 16 10:48:05 pqmon NOTE: 121328     1   78671  2999953856    199999
> 3         0     46656 65902
> Jan 16 10:53:05 pqmon NOTE: 119675     2   80323  2998886320    199999
> 3         0   1084600 66202
> Jan 16 10:58:05 pqmon NOTE: 117090     1   82909  2999998176    199999
> 3         0      2336 66502
> Jan 16 11:03:05 pqmon NOTE: 108592     2   91406  2999023600    199999
> 3         0    646872 66802
> Jan 16 11:08:05 pqmon NOTE: 110615     2   89383  2999951776    199999
> 3         0     43720 67102
> Jan 16 11:13:05 pqmon NOTE: 109738     1   90261  2999083032    199999
> 3         0    917480 67402
> Jan 16 11:18:05 pqmon NOTE: 104722     1   95277  2999991648    199999
> 3         0      8864 67702
> Jan 16 11:23:05 pqmon NOTE:  95543     1  104456  2990078032    199999
> 3         0   9922480 68002
> Jan 16 11:28:05 pqmon NOTE: 101960     1   98039  2999493368    199999
> 3         0    507144 68302
> Jan 16 11:33:05 pqmon NOTE: 102930     2   97068  2999302816    199999
> 3         0    440632 68602
> Jan 16 11:38:05 pqmon NOTE: 103688     1   96311  2999486352    199999
> 3         0    514160 68902
> Jan 16 11:43:05 pqmon NOTE: 104502     1   95497  2999266448    199999
> 3         0    734064 69202
> Jan 16 11:48:05 pqmon NOTE: 103715     1   96284  2999853168    199999
> 3         0    147344 69502
> Jan 16 11:53:05 pqmon NOTE: 109279     1   90720  2997999016    199999
> 3         0   2001496 69802
> Jan 16 11:58:05 pqmon NOTE: 104288     1   95711  2999995952    199999
> 3         0      4560 70102
> Jan 16 12:03:05 pqmon NOTE: 103716     2   96282  2999942152    199999
> 3         0     57992 70402
> Jan 16 12:08:05 pqmon NOTE: 102451     1   97548  2999876408    199999
> 3         0    124104 70702
> Jan 16 12:13:05 pqmon NOTE: 105721     1   94278  2999718104    199999
> 3         0    282408 71002
> Jan 16 12:18:05 pqmon NOTE: 102483     1   97516  2983254456    199999
> 3         0  16746056 71302
> Jan 16 12:23:05 pqmon NOTE: 110753     1   89246  2999470920    199999
> 3         0    529592 71602
> Jan 16 12:28:05 pqmon NOTE: 112588     1   87411  2999854912    199999
> 3         0    145600 71902
> Jan 16 12:33:05 pqmon NOTE: 111663     2   88335  2998686896    199999
> 3         0   1114608 72202
> Jan 16 12:38:05 pqmon NOTE: 105013     1   94986  2999977664    199999
> 3         0     22848 72502
> Jan 16 12:43:05 pqmon NOTE: 109028     1   90971  2992194576    199999
> 3         0   7805936 72802
> Jan 16 12:48:05 pqmon NOTE: 108982     1   91017  3000000296    199999
> 3         0       216 73102
> Jan 16 12:53:05 pqmon NOTE: 110827     1   89172  2999880008    199999
> 3         0    120504 73402
> Jan 16 12:58:05 pqmon NOTE:  99670     1  100329  2999526936    199999
> 3         0    473576 73702
> Jan 16 13:03:05 pqmon NOTE: 102935     1   97064  2999980528    199999
> 3         0     19984 74002
> Jan 16 13:08:05 pqmon NOTE: 105077     0   94923  3000000512    199999
> 3         0         0 74302
> Jan 16 13:13:05 pqmon NOTE:  99958     2  100040  2999449840    199999
> 3         0    313192 74602
> Jan 16 13:18:05 pqmon NOTE: 103311     1   96688  2995839624    199999
> 3         0   4160888 74902
> Jan 16 13:23:05 pqmon NOTE: 108959     2   91039  2999982584    199999
> 3         0     17768 75202
> Jan 16 13:28:05 pqmon NOTE: 112787     1   87212  2998769720    199999
> 3         0   1230792 75502
> Jan 16 13:33:05 pqmon NOTE: 115691     2   84307  2999848216    199999
> 3         0    147784 75802
> Jan 16 13:38:05 pqmon NOTE: 120612     1   79387  2999625864    199999
> 3         0    374648 76102
> Jan 16 13:43:05 pqmon NOTE: 115633     2   84365  2999669816    199999
> 3         0    323160 76402
> Jan 16 13:48:05 pqmon NOTE: 119396     2   80602  2999536544    199999
> 3         0    463552 76702
> Jan 16 13:53:05 pqmon NOTE: 126014     2   73984  2999581264    199999
> 3         0    419008 77002
> Jan 16 13:58:05 pqmon NOTE: 117288     2   82710  2999921400    199999
> 3         0     76456 77302
> Jan 16 14:03:05 pqmon NOTE: 120417     2   79581  2998780160    199999
> 3         0    968264 77602
> Jan 16 14:08:05 pqmon NOTE: 121814     1   78185  2999724368    199999
> 3         0    276144 77902
> Jan 16 14:13:05 pqmon NOTE: 114444     2   85554  2999958744    199999
> 3         0     41336 78202
> Jan 16 14:18:05 pqmon NOTE: 100704     1   99295  2999990792    199999
> 3         0      9720 78502
> Jan 16 14:23:05 pqmon NOTE:  99111     2  100887  2999486480    199999
> 3         0    513456 78802
> Jan 16 14:28:05 pqmon NOTE:  90364     1  109635  2998573848    199999
> 3         0   1426664 79102
> Jan 16 14:33:05 pqmon NOTE:  93249     1  106750  2999982296    199999
> 3         0     18216 79402
> Jan 16 14:38:05 pqmon NOTE:  93826     2  106172  2997471920    199999
> 3         0   2143856 79702
> Jan 16 14:43:05 pqmon NOTE:  94575     1  105424  2999616000    199999
> 3         0    384512 80002
> Jan 16 14:48:05 pqmon NOTE:  93327     2  106671  2975330760    199999
> 3         0  24156600 80302
> Jan 16 14:53:05 pqmon NOTE: 110816     1   89183  2988839504    199999
> 3         0  11161008 80602
> Jan 16 14:58:05 pqmon NOTE: 111887     2   88111  2999588728    199999
> 3         0    370608 80902
> Jan 16 15:03:05 pqmon NOTE: 113089     3   86908  2998917248    199999
> 3         0    834984 81202
> Jan 16 15:08:05 pqmon NOTE: 114215     1   85784  2981162640    199999
> 3         0  18837872 81502
> Jan 16 15:13:05 pqmon NOTE: 117087     1   82912  2999889272    199999
> 3         0    111240 81802
> Jan 16 15:18:05 pqmon NOTE: 120026     0   79974  3000000512    199999
> 3         0         0 82102
> Jan 16 15:23:05 pqmon NOTE: 122294     2   77704  2999957328    199999
> 3         0     33592 82402
> Jan 16 15:28:05 pqmon NOTE: 126456     3   73541  2999954968    199999
> 3         0     36280 82702
> Jan 16 15:33:05 pqmon NOTE: 128278     1   71721  2999999696    199999
> 3         0       816 83002
> Jan 16 15:38:05 pqmon NOTE: 134706     2   65292  2999999344    199999
> 3         0       848 83302
> Jan 16 15:40:43 pqmon NOTE: Exiting

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: GXL-914371
Department: Support LDM
Priority: Normal
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.