Error reading netCDF file

[Rick Grubin <grubin@xxxxxxxx>]

I have an OPeNDAP-enabled client (NCL  http://www.ncl.ucar.edu) that is  
trying to read the following file:

    http://iridl.ldeo.columbia.edu/SOURCES/.NOAA/.NCDC/.GDCN/.lat/dods

The client successfully accesses the OPeNDAP server and retrieves dds  
and das information, but fails when retrieving variable information:

    ncvarget: Error in getting the data: Unknown error 1001

This particular data file has one dimension, its value is 32857 (number  
of stations).  The two variables contained within the file (ISTA, lat)  
are dimensioned with this value.  Interestingly, dncdump can  
access/read the file without difficulty.

I'm successful when accessing/reading a subsetted data file of the  
larger, overall file (sorry for the weird URL):

     
http://iridl.ldeo.columbia.edu/expert/SOURCES/.NOAA/.NCDC/.GDCN/ 
lat%286.666659%29%2836.66666%29masknotrange/SELECT/lat/%28 
-40.08333%29%28-11.66666%29masknotrange/SELECT/.lat/dods

In the course of debugging my OPeNDAP-enabled client, I find that the  
error noted above occurs in the following calling sequence:
    ncvarget ---> nc_get_vara ---> DODvario ---> Connect::request_data  
--->
        HTTPConnect::fetch_url ---> HTTPConnect::plain_fetch_url --->  
HTTPConnect::read_url

The url being read at the time of failure is:

     
http://http://iridl.ldeo.columbia.edu/expert/SOURCES/.NOAA/.NCDC/.GDCN/ 
.lat/dods.dods?ISTA%5b0%3a1%3a32856%5d

It appears that the OPeNDAP library code is attempting to read all  
32857 values for the variable ISTA (0 --> 32856) at a time;  this  
generates the error noted above.

Interestingly, dncdump successfully reads the entire file (first URL  
noted above) every time.  Note that dncdump does not try to read all  
values for the variable ISTA at once; rather, it breaks up its  
retrieval into 1000-element chunks.  As an example, here's a URL that  
dncdump is using (at the same end point (HTTPConnect::read_url) in the  
calling sequence noted above):

    
http://http://iridl.ldeo.columbia.edu/expert/SOURCES/.NOAA/.NCDC/.GDCN/ 
.lat/dods.dods?ISTA%5b2000%3a1%3a2999%5d

   (reading elements 2000 -- 2999)

It would seem that there's an upper limit on how much data can be  
retrieved at one time via the OPeNDAP libraries.  True?  I base this  
assumption on the fact that an OPeNDAP-enabled client can't read a  
relatively large chunk of data at once, whereas dncdump, because it  
breaks up the retrieval of the data into "gulps" (1000 elements at a  
time), always succeeds.

Note that this error is consistently repeatable; I can reproduce it  
every time I run my client.

-Rick.
----
Rick Grubin      NCAR/CISL/SCD/VETS
Visualization + Enabling Technologies
grubin@xxxxxxxx             303.497.1832