Daryl, Couple of quick comments: 500MB queue is small. You are receiving up to 1.2GB per hour on average: http://my.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume1?pircsl4.agron.iastate.edu The trade offs are, smaller queue will fit into available memory better, but the relay to downstreams and pqact processing could benefit from more space. If your only rel;ays are to internal machines, then you can probably get by with 500MB if you split your pqact.conf processing into multiple processes. In the GEMPAK release, the $NAWIPS/ldm/etc/gen_pqact.csh script asks if you want to generate a single or multiple pqact.conf files. Anyhow, it provides the example lines which show how to use multiple pqact.conf files, for example: % $NAWIPS/ldm/etc/gen_pqact.csh Generating pqact.gempak file using: GEMTBL = /home/gempak/GEMPAK5.8.2a/gempak/tables GEMPAK = /home/gempak/GEMPAK5.8.2a/gempak Do you want to combine entries to a single pqact.gempak? [y/n] n ################################################################ Place the generated PQACT files into ~ldm/etc. Use the following entries for the LDM ldmd.conf file: exec "pqact -f ANY-NNEXRAD-CRAFT-NIMAGE etc/pqact.gempak_decoders" exec "pqact -f WMO etc/pqact.gempak_nwx" exec "pqact -f MCIDAS|NIMAGE etc/pqact.gempak_images" exec "pqact -f NNEXRAD|WSI|FNEXRAD etc/pqact.gempak_nexrad" exec "pqact -f CRAFT -p BZIP2/K[A-D] etc/pqact.gempak_craft" exec "pqact -f CRAFT -p BZIP2/K[E-K] etc/pqact.gempak_craft" exec "pqact -f CRAFT -p BZIP2/K[L-R] etc/pqact.gempak_craft" exec "pqact -f CRAFT -p BZIP2/K[S-Z] etc/pqact.gempak_craft" ---------------------------------------------------------------------------------- The above is particularly useful when FILE'ing every Level II and Level III radar product. The 4 pqact's for craft were done to balance the 32 open streams among the 124 radars which may not be necessary....but it is an example of how to use the same pqact.conf file with different subsets of the data stream pattern. The number of level III products is actually quite alot- and really a load when people try to uncompress the zlib products in realtime (which I wouldn't recommend). The newer LDM releases will allow more streams from pqact, which is 1024 on Linux, whereas it had been 32, but pqact can fall behind if it is heavily loaded eg with slower decoders. Issuing a "kill -USR2" twice to the pqact process will put the program in debug mode and output "Delay:" messages that show how long it took to process something once it arrived in your local queue. Generally this should be down under a few seconds. Hint...don't leave in debug mode too long, since this will slow things down too- issua another "kill -USR2" to go back to silent. If the Delay ever gets to the age of the oldest product in your queue (shown by "pqmon") then your data will overwrite before pqact acts on it. Steve Chiswell Unidata User Support >From: Daryl Herzmann <address@hidden> >Organization: UCAR/Unidata >Keywords: 200505101824.j4AIO8P3026338 >Hi Steve, > >Wow, what a reply! Please see my comments below. > >On Tue, 10 May 2005, Unidata Support wrote: > >> Your laptop address was either strange or not on the list, so I think >> that is why it bounced. > >Hmmm, wonder if my IP is not resolving? I am using IMAP locally. > >> Anyhow, you can see what we received here for the 00Z GFS: > >Good, the problem is just at my end! > >> The above shows that Pircsl4 is only getting about 18GB perday, and very >> consistent in that amount, so likely not any temporary network trouble >> etc., and nothing new in the pattern, so it seems very well behaved. I >> see your request pattern is: (RUC2/#252 |MT.(avn|gfs).*DF.gr1) That >> would account for the volume being less than what we have to send you! > >Good. > >> That means, the data is getting to you in good time. As a result, if you >> are not seeing all the data on your disk, then likely your pqact >> proccess is falling behind so that the data is getting scoured out of >> your queue before you get it processed. If necessary, I can send you >> some code for your pq.c routine that will flag when pqact falls behind. > >That must be it. I was running 6.2.1 and just upgraded to 6.3.0 to make >sure I am current. My queue is 500 MB, so that should be okay. > >> So...things to look for: Are you running multiple pqact processes, or a >> single? > >single. > >> Splitting up the pqacts shares the load so they can process faster. > >Wow, I have never heard of doing this. That is an interesting idea! > >> Are you decoding the data in that pqact.conf file too, or just >> FILE'ing as gribs as shown below? > >Just FILE > >> Since it is the 00Z f000 time, it might be that your system is >> particularly busy around this time, so IO is the bottleneck (check > >This was one of the machine I was complaining about on the ldm list. >RHEL3 sucks. I am have upgraded to the U5 beta kernel and see if that >helps. Going to RHEL4 will happen if this doesn't help. > >> Let me know if you have any questions. > >Thanks for the help. Hope this email doesn't bounce too... > >daryl > >-- >/** > * Daryl Herzmann (address@hidden) > * Program Assistant -- Iowa Environmental Mesonet > * http://mesonet.agron.iastate.edu > */ > -- NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.