[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20030214: PDINRS->LDM


Stonie,

The FOS wrapper on the IDD products uses the last three digits of the SBN
sequence. Pqing was modified a while back to compute the checksum for
WMO and NNEXRAD feed products skipping the first 11 bytes
(eg SOH \r \r \n SP # # # \r \r \n ), so that the sequence number was not used
in the checksum which allowed detection of the rebroadcast duplicates
(especially in NEXRAD products) where only the sequence number varied.

I modified my ingestor similarly to the pqing work around as well.
I use the entire SBN sequence number in the product queue identification
metedata for tracking down inconsistencies (the SDI boxes just use the MOD 1000
sequence)...but thats not part of the product data....just the metadata.

As you mentioned previously, tacking on the FOS trailer (\r \r \n ETX)
is somewhat dependent upon whether the product already has some of that.
I do a check and tack on the minimum number of bytes to complete the trailer.
The SDI box always tacks on all 4 bytes....which lead to a problem with
the difference in DDPLUS vs HDS feed usage in pqing. Traditionally,
the PPS|DDS|IDS feeds were assumed to be ascii, so the check was
only on SOH and ETX, whereas the HDS feed used the complete
4 byte SOH \r \r \n and \r \r \n ETX sequences. As a reqult, the
use of pqing with the scan_wmo() routine would chop off the final 4 bytes
when the product already had a ETX at the end....thus changing the
MD5 and product length. In the LDM 5.2 release, we changed pqing to
always use scan_wmo_binary() for the combined feedtype "IDS|DDPLUS"
(as well as the HDS feed) in order to remove the source of duplication
in the data stream we were seeing from the SSEC code.

I did comparisons vs the SDI ingestor which backends to pqing, and my ingestor
and generally found about 7 to 50 products an hour difference in the feeds
(most due to suspected bugs in he SDI software mangling certain ETA products)
and some hydro products from KWOH that have an embedded FOS termination sequence
that pqing trips on.

Chiz

On Tue, 18 Feb 2003, Stonie R. Cooper wrote:

> If I may prematurely advocate a path, considering that we are trying to
> syncronize IDD nodes with various NOAAPort vendors, I would suggest that if
> FOS style sequencing is continued on IDD, that we use the last three digits
> of the SBN product sequence number.  As NCF assigns the SBN product sequence
> numbers . . . this should give us all a universal reference.  But this
> assumes that FOS style product sequencing is even needed . . .
>
> I'll await for info from Chiz to begin testing . . . we have a few *.edu
> customers that may allow us to use them for testing . . . such as UGA, LDEO
> (Columbia U), Lyndon, or Embry Riddle.
> - --