[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[CONDUIT #CPI-946647]: LDM: Capturing CONDUIT Feed

Kevin,

The pqing program is designed to read FOS style products with WMO headers. 
Since the CONDUIT data products are not FOS GRIB bulletins like those 
you would find in the HDS feed, you cannot use the pqing program to
read arbitrary products from a file you have pqcat'd out.

The CONDUIT products are simply the GRIB bulletins that you would find on
the NWS ftp servers. You can simply pass those files to your decoder
the same way you would one of those files.

Note that you can have the characters "7777" (or longer runs of 7s) within a 
grib 
message at any  location. To signify the end, the location of the "7777" will 
have 
to be at the location calculated from the length in the IDS.

Steve Chiswell
Unidata User Support



> Hi,
> We have been having problems with some of the GFS Geo Grid 3 files
> coming through the CONDUIT feed on one of our machines.
> They seem to be getting chopped off (an error is returned from XCD
> saying '7777' is not in the right place).
> Anyway, in an attempt to debug this, I have been trying to capture the
> CONDUIT feed, so that I can reinject it on another machine that is not
> having this CONDUIT problem.
> I have been using the command:
> pqcat -f CONDUIT -q /home/ldm/data/ldm.pq test_CONDUIT
> on a queue near to where the error message is occurring. This tells me
> that I have several thousand products captured to test_CONDUIT.
> So, I get a large test_CONDUIT file, transfer it to the other machine
> and then attempt to inject it into the queue.
> pqing -v -l test.log test_CONDUIT
> The output to test.log looks like the following:
> 
> May 17 19:36:05 pqing[26160] NOTE: Starting Up
> May 17 19:36:05 pqing[26160] NOTE: FILE "test_CONDUIT"
> May 17 19:36:05 pqing[26160] INFO: Creating AFOS-like pil tags
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 32768
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 65536
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 131072
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 262144
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 524288
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 1048576
> May 17 19:36:05 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 2097152
> May 17 19:36:05 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 4194304
> May 17 19:36:05 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 8388608
> May 17 19:36:05 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:05 pqing[26160] INFO: Expanding input buffer size to 16777216
> May 17 19:36:06 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:06 pqing[26160] INFO: Expanding input buffer size to 33554432
> May 17 19:36:06 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:06 pqing[26160] INFO: Expanding input buffer size to 67108864
> May 17 19:36:07 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:07 pqing[26160] INFO: Expanding input buffer size to 134217728
> May 17 19:36:09 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:09 pqing[26160] INFO: Expanding input buffer size to 268435456
> May 17 19:36:12 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:12 pqing[26160] INFO: Expanding input buffer size to 536870912
> May 17 19:36:21 pqing[26160] ERROR: Buffer would exceed 1048576,
> resetting input
> May 17 19:36:21 pqing[26160] INFO: Expanding input buffer size to 1073741824
> May 17 19:36:31 pqing[26160] NOTE: Exiting
> May 17 19:36:31 pqing[26160] NOTE:   Queue usage (bytes):-1294966784
> May 17 19:36:31 pqing[26160] NOTE:            (nregions):  396381
> May 17 19:36:31 pqing[26160] NOTE:   Duplicates rejected:       0
> May 17 19:36:31 pqing[26160] NOTE:   WMO Messages seen:         0
> May 17 19:36:31 pqing[26160] NOTE:   SOH/ETX missing  :         0
> May 17 19:36:31 pqing[26160] NOTE:   parity/chksum err:         0
> May 17 19:36:31 pqing[26160] NOTE:   WMO format errors:         0
> May 17 19:36:31 pqing[26160] NOTE:   FILE Bytes read:    1572990822
> 
> I don't know what this means, but I have used a similar procedure with
> HDS and didn't have the above problems.
> Thanks!
> Kevin Baggett, UW-SSEC
> 
> 


Ticket Details
===================
Ticket ID: CPI-946647
Department: Support CONDUIT
Priority: Normal
Status: Closed