[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20000815: Java NETCDF RMI (fwd)


Hi Phoebe:

> 
> Hi, I am working on a distributed data system using your netcdf Java/rmi
> API and find myself unclear about the efficiency issue involved. I
> understand for local netcdf files a buffer is used in
> ucar.netcdf.RandomAccessFile to speed IO operations and probably the same
> mechanism used on the RMI server side. Is there any buffering on the
> client side or each getXXX/putXXX call has to involve a round trip over
> the network? Would copyIn/Out be more efficient for remote access in this
> scenario? Any information would be very appreciated.
> 
> --Best regards
> 
> --Phoebe Zhang


The getXXX/putXXX calls are horrendously slow for both remote and local
access; you _must_ use copyin/copyout for efficient access.  We have a
different, prototype netcdf API that we are experimenting with that
makes this clearer, since it is a common mistake to use the
getXXX/setXXX methods. However we have not yet incorporated the RMI part
into the new API.

Note that you must be running a server to use RemoteNetcdf. We are also
considering other ways to provide remote netcdf files, eg using a web
server instead of an RMI server.

There is currently no client-side buffering. For large files, it can be
hard to decide how to do that: in memory or construct local file caches?
Keep the local file cache or delete when the remote file is closed? 

From your questions, I assume you are interested in efficiency. The
overhead of the RMI is probably small compared to network latency for
small amounts of data transfered (using copyin), not so good for large
data transfer. 

We will also be evaluating new, more efficient Java I/O methods that are
currently under development at Sun. We expect that these will eliminate
the current I/O efficiencies and that Java will be as fast as C for
local access. Those are a year or two off, however.  

We'd like to hear how you want to use remote netcdf files, how big they
are, average data transfer size, latency needs, etc. We are considering
possible new development and your input would be useful to us.