[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20031103: Homegrown LDM pickup . . .



Stonie,

I just allocate a memory heap at the beginning (and grow it if necessary
for the size of the porduct to be received), and open the queue
once at the start.

As the data comes in (T1 rate is pretty slow compared to
 the eventual 45mbit/sec for dvb-s), I update the MD5 chacksum and place in the
memory heap with the offset from the start, and then when the product 
completion 
is received, do the MD5Final, create the product structure information and set 
the 
prod.data pointer to the memory heap location, then use the pq_insert( pq, 
prod) call. 
Its very fast and has no problems on our boxes reading all 4 noaaport channels.
In fact, before the satellites were Zlib compressed, I was running a PNG
compression (again updating the PNG as each portion of the product was
received) with no trouble.

There is one read process for each channel, so 4 processes total (actually
now 3 after GOES-E/W combined). No forking.

We have 2 boxes here running Solaris X86 and one in DC on a SunFire 280R.
I have also been running one here on a multicast stream on the dvb-s
test stream. I think we have a signal problem on DVB-s and have a filter
being installed (I was out of the office for the
workshops so I don't know where we stand on that installation).
If you want to relate experience with the DVB-s power thus far,
it would be welcomed.

Also, please (please, please) feel free to make suggestions on
our web portal design/strategy in the plaza forum since that will 
probably get the added attention of the portal developers and may
carry more weight than we get just from this end. We (developers) have
discussed a Forge-site type of approch as well as other support
tools and will have to move away from our current method soon.
At some point, we have to try to account for what good we are doing
to the NSF, and user numbers are helpful. Perhaps a 1 time guest book
sign in for each download would be possible (thoughts)? Obviously mirror
site downloads will go uncounted, but the relative usefullness of
GEMPAK, McIDAS, IDV etc. versus the bang-for-buck is necessary
for the oversite committees.

Steve







>From: "Stonie R. Cooper" <address@hidden>
>Organization: Planetary Data, Incorporated
>Keywords: 200311031403.hA3E38Ob021964

>Steve,
>
>I must assume you use some flatfile system to "queue" data . . . I just tried 
>your algorithm with four streams of data (NWSTG, GOES, DCP, and DVB-S) over 
>the weekend before I break our antenna down for a move . . . and the streams 
>hopelessly backed up.  And thus my delima . . . in our old paradigm, we 
>simply forked for each incoming product off a new pqinsert wrapper function 
>instance and did the pq_open();pq_insert();pq_close(), but that's where 6.0.x 
>doesn't want to play well.
>
>Have you had success with streaming straight through with what you have done?
>
>Stonie
>
>On Thursday 30 October 2003 18:29, Steve Chiswell wrote:
>> Stonie,
>>
>> I do all work through the libldm.a routines.
>>
>> That is, I allocate a MD5 context and update the signature as the
>> 4-5 Kbyte chunks arrive from the dish (much more efficient to do this
>> for 10+Megabyte satellite images than waiting to
>> use pqinsert program after a full product arrives)
>>
>> Then, when the product is complete, finalize the MD5, and fill in the
>> product structure and insert into the LDM queue using:
>> status = pq_insert(pq, &prod);
>>
>> where pq is the product queue opened at the beginning, and prod is a
>> data structure containing the data, signature, sequence number,
>> produjct identifier, timestamp and product size.
>>
>> Steve
>