I receive a lot of data, and typically my queues are not big enough to
hold an hour's worth of data.
Right now I only have about 10-15 minutes of data in my queues, but I
have not received any duplicated or delayed data.
Alan.Hall wrote:
Generally a larger volume of data and too small of a queue. I'm
assuming a sustained rate of 6-10Mbits for the NEXRAD II feed, that
means to keep an hours worth of data in the queue would take a queue
size of 2.7GBytes to 4.5GBytes. Of course you don't have to have an
hours worth of data in the queue as long as the pqact can process the
data in the queue before they drop out of the queue. By processing I
mean to pass it down stream and pqact.
You can multi-stream the pqacts and gain some performance.
When Dual Pol comes around you'll need a 9 GByte queue and then the
AVSET might add another 20% onto that.
Alan.
Gilbert Sebenste said the following on 7/9/2009 10:40 AM:
On Thu, 9 Jul 2009, Gilbert Sebenste wrote:
Started yesterday, central region for sure, I haven't checked the
other regions yet. Here are my latest files for KLOT
(Romeoville/Chicago)...
-rw-rw-r-- 1 ldm users 2255367 Jul 9 08:58 KLOT_20090709_1322
Notice, for example, the 1322Z data file is twice the normal size
and came in about 40 minutes late.
I also just saw something using pqmon. (Darryl, just saw your message).
I would have to increase my queue size to nearly 6 GB to stay ahead
of this. No, no, and heck, no! I'm trying to run my queue out of
/dev/shm, to which I have 1.3 GB to do so. It says I am only saving
"558" seconds worth of data with 1.2 GB of queue. That's insane!
Even if the products get rejected, I still have to reject them by
getting them first, correct?
OK. So when I use idd.aos.wisc.edu, flood.atmos.uiuc.edu,
idd.unidata.ucar.edu or bigbird.tamu.edu singularly, that is, as the
primary feed with no backups, I don't get problems. But I just tried
it now with
request NEXRAD2 ".*" notam.atmos.uiuc.edu
request NEXRAD2 ".*" bigbird.tamu.edu
In order...and I get the double feed. I tried it in reverse, same
thing (with bigbird as primary). Tried it with other sites...same
results. And when I do an ldmadmin watch, look what I see:
Jul 09 14:36:01 pqutil INFO: 38876 20090709135846.462 NEXRAD2
49042 L2-BZIP2/KJAX/20090709135626/49/42/I/V03/0
Jul 09 14:36:01 pqutil INFO: 30687 20090709133606.451 NEXRAD2
626047 L2-BZIP2/KOAX/20090709133327/626/47/I/V03/0
Jul 09 14:36:01 pqutil INFO: 15952 20090709135846.302 NEXRAD2
507050 L2-BZIP2/KVNX/20090709135452/507/50/I/V03/0
Jul 09 14:36:01 pqutil INFO: 53219 20090709135845.193 NEXRAD2
570006 L2-BZIP2/KGWX/20090709135641/570/6/I/V04/0
Jul 09 14:36:01 pqutil INFO: 23850 20090709133606.815 NEXRAD2
906043 L2-BZIP2/KPUX/20090709133401/906/43/I/V03/0
Jul 09 14:36:01 pqutil INFO: 51674 20090709143600.836 NEXRAD2
533008 L2-BZIP2/KILX/20090709143434/533/8/I/V03/0
Jul 09 14:36:01 pqutil INFO: 18361 20090709133606.928 NEXRAD2
803027 L2-BZIP2/KIWA/20090709133443/803/27/I/V03/0
Jul 09 14:36:01 pqutil INFO: 45660 20090709135845.333 NEXRAD2
57012 L2-BZIP2/KRLX/20090709135810/57/12/I/V03/0
Jul 09 14:36:01 pqutil INFO: 15858 20090709143601.012 NEXRAD2
233018 L2-BZIP2/KBLX/20090709143228/233/18/I/V03/0
Jul 09 14:36:01 pqutil INFO: 21409 20090709135846.469 NEXRAD2
384016 L2-BZIP2/KLVX/20090709135726/384/16/I/V03/0
Jul 09 14:36:01 pqutil INFO: 10574 20090709135846.597 NEXRAD2
87029 L2-BZIP2/KFSD/20090709135708/87/29/I/V03/0
Jul 09 14:36:01 pqutil INFO: 32569 20090709143559.869 NEXRAD2
786017 L2-BZIP2/KINX/20090709143517/786/17/I/V03/0
Jul 09 14:36:01 pqutil INFO: 42332 20090709135847.305 NEXRAD2
701014 L2-BZIP2/KDMX/20090709135804/701/14/I/V03/0
Jul 09 14:36:01 pqutil INFO: 20409 20090709143600.580 NEXRAD2
61033 L2-BZIP2/KRLX/20090709142705/61/33/I/V03/0
Jul 09 14:36:01 pqutil INFO: 38611 20090709135847.733 NEXRAD2
777042 L2-BZIP2/KINX/20090709135647/777/42/I/V03/0
Jul 09 14:36:01 pqutil INFO: 10000 20090709143601.246 NEXRAD2
512014 L2-BZIP2/KARX/20090709143452/512/14/I/V04/0
Jul 09 14:36:01 pqutil INFO: 29056 20090709143559.812 NEXRAD2
689019 L2-BZIP2/KLZK/20090709143510/689/19/I/V03/0
Jul 09 14:36:01 pqutil INFO: 67204 20090709133606.339 NEXRAD2
709013 L2-BZIP2/KIND/20090709133526/709/13/I/V03/0
Obviously delayed duplicates, and I JUST cleared/deleted and remade
the queues as the above was happening, so my queues weren't full.
This never used to happen before. What's going on?
*******************************************************************************
Gilbert Sebenste
********
(My opinions only!)
******
Staff Meteorologist, Northern Illinois
University ****
E-mail:
sebenste@xxxxxxxxxxxxxxxxxxxxx ***
web:
http://weather.admin.niu.edu **
*******************************************************************************
_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
-------------------------------------------
There are 2 kinds of people in the world:
1) Those who can extrapolate from incomplete data.
-------------------------------------------
Karen.Cooper@xxxxxxxx
Phone: 405-325-6982
Cell: 405-834-8559
INDUS Corporation
National Severe Storms Laboratory
