[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20030214: PDINRS->LDM

>From:  "Stonie R. Cooper" <address@hidden>
>Organization:  Planetary Data, Incorporated
>Keywords:  200302141847.h1EIlR607186 LDM PDI NOAAPORT

Hi Stonie,

>Declan indicated that in a conversation with you at the AMS conference, you 
>wanted to see if we could change the streaming into LDM from our software to 
>better match IDD.

To be more specific, I asked Declan if you (PDI) are interested in making
the data coming out of your NOAAPORT ingest systems compatible with
the ingesters currently used in the IDD.  "Compatible" means that the
MD5 checksum for each product would have to match that being calculated
by the other ingesters.

>I would love that.

Excellent.  I was hoping that you would be interested!  FYI, Jeff
Masters of the Weather Underground, Inc. has already started looking
at what it will take to get their ingester(s) working with the IDD.

>Not having access to IDD, I've used UNIDATA docs to try to
>guess the "look" of feeders, extensions, and wrappers . . . but I know from 
>some feedback from *.edu users, that my attempts were probably ill 
>implemented.

We can give you an IDD feed of whatever feed(s) you want/need to accomplish
the task.  We can talk about this in more detail later.

>I don't believe there needs to be any code change . . . within our software, 
>most things are controlled via a config file.  As an example, for GRIB data, 
>we currently configure to stream to the LDM plugin to define the data as HDS, 
>remove the CCB, and have a preamble and closing character 
>(\001\r\r\nnnn\r\r\n at the beginning and \003 at the end).  Another example, 
>RedBook, we currently configure to define the feedtype as NGRAPH, keep the 
>CCB, and to not have a preamble.

Sounds interesting...

>On the LDM plugin side, the currently used app uses the function wrapped 
>pqinsert() [renamed pqinsert_function], that enables the setting of feedtype, 
>MD5, etc.  Is there another approach you guys would prefer?

Chiz is the real expert here, so I don't want to stick my foot in my
mouth by offering advice that I am not qualified to offer :-)

>One problem we have come across . . . and I have hesitated to bother you guys 
>with it . . . is the use of the psuedo FOS sequence number in the fake 
>preamble.
>Jim Cowie, when working for Witi, just had us set it to "999", but then we've 
>had academic customers want it to be sequenced.  Doesn't a sequenced FOS 
>header thingy kind of defeat the MD5 checksum for deduplication?  What would 
>be your preference for a default config . . . as this is an optional thing 
>that is invoked as an argument to the plugin startup in our latest (and not 
>released) version of the plugin.  We could also use the last three digits of 
>the SBN frame sequence number or last three digits of the SBN product 
>sequence number.  If the last three digits of the SBN product sequence number 
>were used by both thelma and any IDD node using our NRS - would that insure 
>correct MD5 checking and deduplication?

Again, I think that Chiz should be in on this discussion.

>Any guideance to make our product better suited to IDD use is much 
>appreciated.

Super! We will talk about this more at a later time.  I just wanted
to get the ball rolling.

Cheers,

Tom
--
+-----------------------------------------------------------------------------+
* Tom Yoksas                                             UCAR Unidata Program *
* (303) 497-8642 (last resort)                                  P.O. Box 3000 *
* address@hidden                                   Boulder, CO 80307 *
* Unidata WWW Service                             http://www.unidata.ucar.edu/*
+-----------------------------------------------------------------------------+