[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

19990923: LDM queue size with NOAAPORT data

Greg,

My queue is 2GB....but I have WSI nids and CONDUIT grids
coming in too so that is really overkill- and pqexpire really
will hit the CPU hard with such a large queue. DDS and PPS
from FOS should be less than 20MB. Most of the FOS size
is with the HRS feed.

The NWSTG part of NOAAPort can be handled with a 150MB queue
(the biggest data hit is the AVN and MRF which come in on
several grid projections).

The GOES channels can fluctuate since the RSO mode will virtually
double the number of GOES-E and GOES-W visible images which currently
have no compression- so thet are all the same 21-26MB each.
I've been working with compresison algorithms for that purpose.

Here are my numbers (I have channel 4 comming in also,
so the DCP data is added to my NWSTG, and the composite images are
added to my GOES numbers- but both are relatively small in comparison)

                    Compressed
GOES-E+W       ###    Bytes         NWSTG          ###     Bytes
19990922034107 95   92261799        19990922035948 16555 48645078
19990922045039 88   75228694        19990922045957 20451 70544314
19990922055039 82   57921665        19990922055959 11214 14568832
19990922065952 93   41295426        19990922065958 15198 124510136
19990922075908 130  38381966        19990922075959 10984 25334223
19990922085835 73   52536829        19990922085959 15781 34551891
19990922095835 53   47093689        19990922095954 12451 20950674
19990922105604 103  91922989        19990922105959 10539 15391046
19990922115836 170 172981927        19990922115950 11229 14562465
19990922125957 129 120406758        19990922125959 13381 18126298
19990922130921 26   34920472        19990922135956 13543 20811009
19990922145835 119 117765510        19990922145959 13686 24577272
19990922155955 118 110756079        19990922155952 16774 41717662
19990922165957 171 160540386        19990922165954 21567 75234532
19990922175834 153 154799243        19990922175956 11704 15460334
19990922185955 134 115402957        19990922185958 12539 17781658
19990922195953 167 151232188        19990922195958 12357 56949049
19990922205835 165 161194065        19990922205951 10816 69446517
19990922215957 123 114192846        19990922215958 13428 21105568
19990922225955 172 166161437        19990922225959 10921 11145268
19990922234147 118 121550005        19990922235904 10897 8621746
19990923015956 135 133612147        19990923005956 12369 11071542
19990923025925 164 170823460        19990923015955 12454 17031358
19990923034108 95   92627804        19990923025959 14496 27845877
19990923045038 44   30618575        19990923035959 16849 53159229
19990923055038 82   58452761        19990923045959 19121 62334174
19990923065955 86   25385613        19990923055959 10906 13800172
19990923075908 130  39325707        19990923065959 14500 121126446
19990923085836 73   53151922        19990923075959 11033 30106875
19990923095835 53   47563165        19990923085959 15781 25704283
19990923105603 112 111322587        19990923095951 11813 15478174
19990923115835 170 172608841        19990923105958 10409 10444999
19990923125955 129 121610422        19990923115959 11092 10683307
19990923135958 173 169677741        19990923125959 12562 11876597
19990923145836 146 155086997        19990923135958 13126 14297919
19990923155954 123 113319434        19990923145959 12693 15580558
19990923165955 162 156779569        19990923155958 17059 41230921
19990923172410 66   73793720        19990923165955 21877 76397048

My compressed images average 40% of the original size, so the peak
GOES 170MB is ~425MB, and the peak for NWSTG is 120MB.

So, I would suggest as a starting point for queue size:
425 + 120 + 20 = 565MB.

Hope this helps.

Steve Chiswell
Unidata User SUpport





>From: Gregory Grosshans <address@hidden>
>Organization: .
>Keywords: 199909231704.LAA08729

>This is a multi-part message in MIME format.
>--------------43CD73ED2FC2D68EF87F840B
>Content-Type: text/plain; charset=us-ascii
>Content-Transfer-Encoding: 7bit
>
>Is there a recommended LDM queue size when ingesting NWSTG, GOES-EAST and
>GOES-WEST
>channels from NOAAPORT?  I increased the queue to 200 MB and I receive log
>messages about
>"Deleting oldest to make space #### bytes".   The particual LDM is also receiv
> ing
>DDS and PPS
>data as well.
>
>Thanks,
>Gregg Grosshans
>
>
>--------------43CD73ED2FC2D68EF87F840B
>Content-Type: text/x-vcard; charset=us-ascii;
> name="grosshans.vcf"
>Content-Transfer-Encoding: 7bit
>Content-Description: Card for Gregory Grosshans
>Content-Disposition: attachment;
> filename="grosshans.vcf"
>
>begin:vcard 
>n:Grosshans;Gregory
>tel;fax:405-579-0700
>tel;work:405-579-0720
>x-mozilla-html:TRUE
>url:www.spc.noaa.gov
>org:DOC/NOAA/NWS/NCEP/Storm Prediction Center
>adr:;;1313 Halley Circle                               ;Norman;OK;73069;
>version:2.1
>email;internet:address@hidden
>x-mozilla-cpt:;0
>fn:Gregory Grosshans
>end:vcard

>--------------43CD73ED2FC2D68EF87F840B--
>