Karen,
I have a decoder I provide in GEMPAK that un-bzips the CRAFT data
which does the same as your decoder, but builds with a distributed
bzip2 library on all the platforms I support.
I do split the pqact processing to 3 pqact processes since it facilitates
each handling about 1/2 of the data (since pqact can only have 32
streams open at a time). Each of the pqact processes uses an identical
pqact.conf file....but runs with a -p optoin to work on site IDs in a range.
Attatched is a binary for dcnexr2 that I provide based on the decoder you
provide (but does have improved buffered read through the system SELECT call,
so that it will stay open for additional pieces of the split product without
continually restarting.
My ldmd.conf file has the following lines (which were based on ranges for 65
radars, but could be updated for more to spread the load to more pqact
processes:
exec "pqact -f CRAFT -p BZIP2/K[A-E]
/local/ldm/etc/GEMPAK/pqact.gempak_craft"
exec "pqact -f CRAFT -p BZIP2/K[F-L]
/local/ldm/etc/GEMPAK/pqact.gempak_craft"
exec "pqact -f CRAFT -p BZIP2/K[M-Z]
/local/ldm/etc/GEMPAK/pqact.gempak_craft"
My pqact.gempak_craft looks like:
CRAFT
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9]
[0-9])
PIPE decoders/dcnexr2 -s \1 -d /dev/null data/craft/\1/\1_\2_\3
This uses the dcnexr2 binary I have attatched. The -s option writes the 4 letter
station ID to the unused bytes of the levelII archive2.0 header so that the
data can identify which radar it came from.
I am out of town currently. If you need more info on balancing the pqact
processing, or the buffered reads through the system select mechanism,
I can help on my return Friday.
Steve Chiswell
On Wed, 14 Jan 2004, Steve Emmerson wrote:
> Karen,
>
> > To: address@hidden
> > From: "Karen Cooper" <address@hidden>
> > Subject: pqact PIPE problems
> > Organization: National Severe Storms Laboratory
> > Keywords: 200401141757.i0EHvIp2008899 LDM
>
> The above message contained the following:
>
> > Running LDM 6.0.14 on Linux (RedHat) Workstation 3. I am getting 104
> > NEXRD2 (Level II radar data) feeds from OU. I can receive all of them
> > and write them out to files with no problem.
> >
> > However, if I try, in pqact, to PIPE them through to Harry Edmon's
> > nexradII_bz program for decompression I get error messages which
> > increase in number depending on how many radars I am trying to
> > decompress.
>
> What, exactly, are the error messages?
>
> > (I've written to Harry asking if he has any ideas about
> > what could be causing the problem).
>
> I know nothing about the "nexradII_bz" LDM decoder. Hopefully, either
> Chiz or Tom will be able to help.
>
> > I can get ~20 radars and only see this error occassionally, but if I up
> > it to 30 radars, the errors become almost constant and if I do all
> > 104..... the system locks up. Best I can tell, the decompression alg is
> > trying to do a read of the data in the pipe, but it is empty. This is
> > not an error message, as the read function is returning "0" indicating
> > it read 0 bytes.
>
> The getbuf() function also returns 0 on an error or a (10 minute) timeout.
>
> > I was wondering if there was possibly some limit on the number of pipes
> > that could be open, and/or if anyone else has had a simliar problem with
> > having many pipes open at once.
>
> There is a limit on the number of open pipes: it is reached when the
> pipe() function returns EMFILE in the function pipe_open() of the module
> pqact/filel.c. When this limit is reached, the system-error message
> "pipe" should be logged and the least-recently-used pipe will be closed
> in order to open a new pipe. This could be the source of your problem
> if the maximum number of open pipes on your system is too small.
>
> You should be able to check the limit via the command "ulimit -a" (which
> is a shell built-in). On our Linux 2.4.22 (Fedora Core release 1)
> system, the limit is 1024, which should be more than sufficient.
>
> Please let us know what you discover.
>
> > BTW, we do not use the "-close" option in the PIPE because the NEXRD2
> > data arrives in chunks until a volume scan is complete.
> >
> > --
> > address@hidden
> > Phone:405-366-0434
> > Cell:405-834-8559
> > SAIC/Systems Analyst
> > National Severe Storms Laboratory
>
> Regards,
> Steve Emmerson
>Attachment:
dcnexr2
Description: dcnexr2