[netcdf-java] Some NetCDF-Java Issues in handling data via OPeNDAP

• To: netcdf-java@xxxxxxxxxxxxxxxx
• Subject: [netcdf-java] Some NetCDF-Java Issues in handling data via OPeNDAP
• From: "H. Joe Lee" <hyoklee@xxxxxxxxxxxx>
• Date: Fri, 16 Mar 2012 12:37:47 -0500

Hi,

  I'd like to summarize some issues that I've found in NetCDF-Java in
handling data iva OPeNDAP. I'm not an expert in NetCDF-Java so please
correct me if my observations are wrong.

1. It reads DDS, DAS outputs and then attempts to read the contents of
all string data variables in DDS. Since some strings may be very long,
it degrades the performance.  I don’t know why it tries to access
string variables when it opens data via OPeNDAP. Reading DDS and DAS
seems sufficient.

2. If a string size  is more than 32767, it throws an error. I don’t
understand why it put such restriction yet. The line 151 and 152 of
[1] checks Short.MAX_VALUE, which is 2^15 - 1 = 32767 according to
[2]. This restriction makes it fail to access otherwise valid OPeNDAP
data.

[1] 
https://github.com/Unidata/thredds/blob/master/opendap/src/main/java/opendap/dap/DString.java
[2] http://docs.oracle.com/javase/6/docs/api/java/lang/Short.html#MAX_VALUE

3. It forms a very long URL string if data file has many variables
with long variable names. If the requested URL is too long for server
to handle, it returns an error “opendap.dap.DAP2.Exception: Method
failed:HTTP/1.1 400 Bad Request on URL=http...<all variables will be
listed here>”. I don’t understand why it tries to append all variables
to get DDS output. Here's the part of code that constructs a very long
URL:

// read the data
    DataDDS dataDDS;
    Map<DodsV, DodsV> map = new HashMap<DodsV, DodsV>(2 *
reqDodsVlist.size() + 1);
    if (reqDodsVlist.size() > 0) {

      // Create the request
      StringBuilder requestString = new StringBuilder();
      for (int i = 0; i < reqDodsVlist.size(); i++) {
        DodsV dodsV = reqDodsVlist.get(i);
        requestString.append(i == 0 ? "?" : ",");
        // requestString.append(makeDODSname(dodsV));
        requestString.append(dodsV.getEncodedName());
      }
      String s = requestString.toString();

      try {
        dataDDS = readDataDDSfromServer(requestString.toString());
        root = DodsV.parseDataDDS(dataDDS);

      } catch (Exception exc) {
        logger.error("ERROR readDataDDSfromServer on " + requestString, exc);
        throw new IOException(exc.getMessage());
      }


By the way, this problem can be resolved by editing Tomcat server's
conf/server.xml in Hyrax. By default, it’s 8K. You need to increase it
explicitly by specifying maxHttpHeaderSize.

    <Connector port="8080" maxHttpHeaderSize="65536" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

--
1.217.531.6125@61820-7059

• Follow-Ups:
  • Re: [netcdf-java] Some NetCDF-Java Issues in handling data via OPeNDAP
    • From: Dennis Heimbigner