[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Unidata Support: 20020822: netcdf2All -- HTTPClient package



Roy Britten wrote:


Hi John,

The NetCDF Java (version 2) User's Manual doesn't contain detail of how
NetcdfFiles are created and managed over an HTTP connection. Can you
point me to any information / documentation? I'm assuming that it's not
simply a case of reading the file into memory and working on it from
there.
its extemely simple: instead of a RandomAccessFile, the library opens a HTTPRandomAccessFile. this uses the HTTP "Range" command to get ranges of bytes from the remote file. everything else is exactly the same, and no optimizations like caching are done, although i think buffering is done. 

see ucar.netcdf.NetcdfFile(URL url) and HTTPRandomAccessFile.java

sorry the docs are minimal, an unreleased version has the folowing:
"For HTTP access, the URL should begin with http: for example: http://www.unidata.ucar.edu/staff/caron/test/mydata.nc. For HTTP access, the library uses the HTTP "Range" command to get ranges of bytes from the remote file. The efficiency of the remote access depends on how the data is accessed. Reading large contiguous regions of the file should generally be good, while skipping around the file and reading small amounts of data will be poor. In that case you would be better copying the file to a local drive, or putting the file into a DODS server which will more efficiently subset the file on the server, see DODSNetcdfFile, below. For DODS access, the URL should begin with dods: dods://thredds.unidata.ucar.edu/dodsC/test.nc." 




The problem I discussed with you last week does not occur when the
NetCDF files are located on another server (Apache/1.3.26 instead of
Apache Tomcat/4.0.1). Unfortunately at the moment we are constrained to
using the Tomcat server.

I note that a small NetCDF file (484 bytes) is acquired with one HEAD
and one GET request, acquisition of a larger file (23100 bytes) fails
after one HEAD and two GET requests with the null pointer exception:

Exception in thread "main" java.lang.NullPointerException
        at ucar.netcdf.Attribute.<init>(Attribute.java:119)
        at
ucar.netcdf.NetcdfFile.readV1AttributeArray(NetcdfFile.java:771)
        at ucar.netcdf.NetcdfFile.readV1VarArray(NetcdfFile.java:1611)
        at ucar.netcdf.NetcdfFile.readV1(NetcdfFile.java:1707)
        at ucar.netcdf.NetcdfFile.<init>(NetcdfFile.java:188)
        at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:69)
        at TestConnection.main(TestConnection.java:18)
it looks like there is an earlier failure that only shows up here. hard to debug without an example i can recreate.
obvious guess is a problem with Tomcat range command, but it may be some minor thing (or an error we arent catching) that is unrelated. 




None of the files in our prototype system are (currently) very large. If
the HTTPClient library cannot for some reason reliably communicate with
a Tomcat server, or vice-versa, an alternative for us would be to create
a NetcdfFile from a URLConnection's InputStream. From your knowledge of
the NetCDF Java library can you say how complicated you would think
development along that path would be?
you cant get random access with an InputStream, in effect you would copy the file and access locally, not very efficient for large files, but ok for small.