Tim,
> ... If I understand things correctly,
> there is an RPC call for _every_ product sent down the line. While this
> makes sense for large products, it seems like we are essentially doubling
> the communication overhead for the smaller products. Is there a way that
> the data can be held at the upstream host (maybe at the root host even),
> disseminated in larger bundles to the downstream host, where the data
> stream could be unbundled? The bundles could be defined either in terms of
> number of products or duration of time... where the later would seem to
> make more sense.
Yes, there's even a name for this: batched RPC. A previous version of
the LDM, version 4, used batched RPC to try to minimize communication
overhead. Here's what the Site Manager's Guide said about this:
Previous versions of the LDM protocol relied on batched RPC. This is
simply a remote procedure call that is made over a reliable
connection and expects no return value. The advantage to this system
is that it allows asynchronous processing to occur and the system has
more freedom to use buffering to increase performance. But using
batched RPC over wide area networks caused some problems, so we no
longer use batched RPC in LDM version 5; all the remote procedure
calls have a return value.
Setting the "way-back machine", I seem to remember the problems with
this mechanism included:
- It was not possible to get back a status for whether individual
products were successfully delivered, so the granularity of failure
handling increased and the reliability of the LDM suffered as a
result.
- Batching interfered with timely delivery of products. If there is
a period in a product stream during which no data is injected, the
data that is still batched is "stuck in the pipe" so may not be
delivered as soon it is available, increasing product latencies.
There are ways to solve this with timeouts, but they complicate the
protocol and architecture of the LDM.
- Some vendors implementations of batched RPC was buggy, and we had
to find workarounds for the bugs.
- Using the "rpcgen" tool to generate RPC stubs doesn't work for
batched RPC, so it all has to be hand-coded, which is difficult and
error-prone.
--Russ
_____________________________________________________________________
Russ Rew UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx http://www.unidata.ucar.edu
