[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #SYD-491615]: TDS java exception "java.lang.NegativeArraySizeException" for large netcdf



Well, don't put a nail in that coffin yet, I didn't like how the netcdf 
subset service (ncss) was handling the dataset...I foresee some work on 
that service.

Here's what I'm using:

apache-tomcat 7.0.53 (http://tomcat.apache.org/download-70.cgi)
jdk-7u55-windows-x64 
(http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html)

address@hidden ~/jarfiles/toolsui
$ java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)

-Lansing

On 4/30/2014 3:19 PM, Luke Sheneman wrote:
> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
> for large netcdf
>
> Okay, thanks Lansing.   We will take a look using the TooksUI.   If your
> dev environment running TDS 4.3.21 but it can successfully serve these
> larger netcdf files, then it is definitely a configuration issue on our
> end.   It sounds like you can serve these files on your debug TDS server.
>
> I can open and view the larger netcdfs using Panoply.
>
> Your screen grab looks right.  That "bleeding over" into Canada is
> intentional to capture portions of the Columbia River basin that reside
> in Canada.
>
> Which version of Java are you using in your test environment?  Are you
> using Oracle or OpenJDK java?
>
> We are using Oracle, 64-bit:
>
> address@hidden sheneman]# /usr/java/jdk1.7.0_51/bin/java -version
> java version "1.7.0_51"
> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
>
> Which version of Tomcat?   We are using Tomcat 6.0.24
>
> Hmmm...
>
>
> Thanks,
>    -Luke
>
>
> --
>
> Luke Sheneman, Ph.D
> Technology and Data Services Manager
> Northwest Knowledge Network (NKN) / University of Idaho
> http://www.northwestknowledge.net
>
> address@hidden
> Office: 208.885.4228   Mobile: 208.669.2248
>
>
> On 4/30/14, 1:43 PM, Unidata THREDDS Support wrote:
>> Luke,
>>
>> So here's what I've done to try to capture your problem:
>>
>> 1) I've been running our current development branch out of my debugger,
>> serving one of the files I grabbed.  The development branch is a little
>> ahead of the stable release branch, so I also
>> 2) have been running the latest stable release of the TDS (4.3.21) from
>> the command line, using the same file and catalog.
>> 3) I've also been using both the 4.3.21 and current development branch
>> of ToolsUI to look at the file and interact with the TDS in a live
>> environment
>>
>> What I've found is that neither TDS generates a stack trace when serving
>> up the large .nc file through OPeNDAP.  I can both look at ASCII (as
>> long as I restrict the request) and download a .dods file.  Using
>> ToolsUI, I can hit the TDS remotely and interact with the dataset.  To
>> that end, I set up a viewer to scroll through time and make a movie of
>> the data - looks like continental US plus a little something bleeding
>> over into Canada near the WA/ID/MT border (see attached screen grab).
>>
>> Do you have ToolsUI?  If not, you should grab the jar file here:
>>
>> https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp
>>
>> It's a handy file exploration tool.  You can run it from the command
>> line using:
>>
>> java -Xmx1g -jar toolsui-4.3.jar
>>
>> Once you have it running, you can navigate to either of two tabs to take
>> a quick look at the file.  The first is the NCDump tab, the second is
>> the Iosp/HDF5/HDF5-Objects tab.  The NCDump output will be obvious.
>> When you open under HDF5, selecting anything in the upper window will
>> cause the lower window to populate with relevant information.
>>
>> If the file comes up fine there, then try the THREDDS tab.  In the
>> catalog URL entry field, enter the address of your thredds server,
>> substituting .xml for .html.  For instance, it looks like this for me
>> (attached). Clicking the Show Dataset button should take you to another
>> pane under FeatureTypes/Grids.  From there, you can select the Daily
>> Mean Near-Surface Wind and click the RedRaw, the little red martian in
>> the upper right.  A graphics display will come up, empty, and if you
>> click RedRaw again, you should see the data populating the screen...
>>
>> On the other hand, if something fails quickly, then I'm going to have to
>> fall back on the configuration issue, looking hard at 32 vs. 64 bit
>> libraries somewhere.
>>
>> By the way, what are the other netcdf tools you mentioned you were using
>> that had no problems looking at the files?  As an aside, Matlab I think
>> uses the netcdf-c libraries, and it might be good to make sure it is
>> using a recent set.
>>
>> Fortunately, if this is a NetCDF issue, I have some friends around the
>> corner who can help us out.
>>
>> -Lansing
>>
>>
>> On 4/30/2014 7:27 AM, Luke Sheneman wrote:
>>> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
>>> for large netcdf
>>>
>>> Katherine,
>>>
>>> Let's see what Lansing says.   THREDDs should be able to handle these
>>> files.  The fact that it seems to break when these files are large
>>> indicates either a serious bug in THREDDs or a problem with our
>>> operating environment.   I am curious if unidata sees this java
>>> exception when they try to serve these files on their TDS
>>> installations?   If so, then it is most likely a bug with TDS itself.
>>> That is my expectation, and should be really easy to test.   I know that
>>> Lansing already downloaded some of the netcdfs.
>>>
>>> -Luke
>>>
>>> --
>>>
>>> Luke Sheneman, Ph.D
>>> Technology and Data Services Manager
>>> Northwest Knowledge Network (NKN) / University of Idaho
>>> http://www.northwestknowledge.net
>>>
>>> address@hidden
>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>
>>>
>>> On 4/29/14, 10:10 AM, Hegewisch, Katherine (address@hidden) wrote:
>>>> Luke and Lansing,
>>>>
>>>> Yes, these netcdf files are for the years 1950 - 2100. So those years 
>>>> 2086-2100 you saw in the calendar dates are correct. The years are also in 
>>>> the file names. These are future climate runs.
>>>>
>>>> Yes these are netcdf4 files. I generate them in MATLAB using
>>>> ncid = netcdf.create(FULLFILENAME,'NETCDF4');
>>>>
>>>> Yes these files are compressed. I compress them in MATLAB using
>>>>       netcdf.defVarDeflate(ncid,dataID,true,true,5);
>>>> (the compression is a 5 out of 10... so this should be about 1/2 
>>>> compression).
>>>>
>>>> I wonder if you need me to make smaller files by using smaller ranges of 
>>>> years.
>>>> Luke - would you like me to make an uncompressed file to see if that works 
>>>> better with THREDDS?
>>>>
>>>> Katherine
>>>> ________________________________________
>>>> From: Sheneman, Lucas (address@hidden)
>>>> Sent: Monday, April 28, 2014 7:22 PM
>>>> To: address@hidden
>>>> Cc: Hegewisch, Katherine (address@hidden)
>>>> Subject: Re: [THREDDS #SYD-491615]: TDS java exception 
>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>
>>>> Lansing and Katherine -
>>>>
>>>> Katherine, can you address Lansing's questions about your netcdf
>>>> files?   We are trying to debug thredds.   It works on your smaller
>>>> files, but some things fail on your larger files (>2GB).
>>>>
>>>> Please copy me on your correspondence...
>>>>
>>>> Thanks,
>>>>       -Luke
>>>>
>>>> --
>>>>
>>>> Luke Sheneman, Ph.D
>>>> Technology and Data Services Manager
>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>> http://www.northwestknowledge.net
>>>>
>>>> address@hidden
>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>
>>>>
>>>> On 4/26/14, 8:15 AM, Unidata THREDDS Support wrote:
>>>>> Compressed how?  I can open the file in ToolsUI, another netCDF-Java
>>>>> interface.  If they are compressed somehow, is the 3.1G the compressed
>>>>> or uncompressed size?  Are they actually netcdf4 files?
>>>>>
>>>>> -Lansing
>>>>>
>>>>> On 4/25/2014 6:10 PM, Luke Sheneman wrote:
>>>>>> New Client Reply: TDS java exception 
>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>
>>>>>> Thanks Lansing.
>>>>>>
>>>>>> I believe the netcdfs are compressed.   Perhaps that is playing a role 
>>>>>> here?
>>>>>>
>>>>>> I look forward to seeing what you come up with.
>>>>>>
>>>>>> Thanks,
>>>>>>         -Luke
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Luke Sheneman, Ph.D
>>>>>> Technology and Data Services Manager
>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>> http://www.northwestknowledge.net
>>>>>>
>>>>>> address@hidden
>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>
>>>>>>
>>>>>> On 4/25/14, 3:10 PM, Unidata THREDDS Support wrote:
>>>>>>> Luke,
>>>>>>>
>>>>>>> I finally got my system back online today.  It's not clear what
>>>>>>> happened, but I had to wipe out and reinstall my standard toolkit
>>>>>>> programs.  I fired up a catalog with your file in it, which is 3.1G in
>>>>>>> size.  I noticed that trying to download the entire dataset through the
>>>>>>> ncss service failed because the maximum size limit in threddsConfig.xml
>>>>>>> is set at the default 52428800 (50 Megabytes).  I upped the maximum size
>>>>>>> by editing this parameter in threddsConfig:
>>>>>>>
>>>>>>>           <NetcdfSubsetService>
>>>>>>>             <allow>true</allow>
>>>>>>>             <scour>15 min</scour>
>>>>>>>             <maxAge>30 min</maxAge>
>>>>>>> <!--maxFileDownloadSize>2097152</maxFileDownloadSize-->
>>>>>>> <!--maxFileDownloadSize>52428800</maxFileDownloadSize-->
>>>>>>> <maxFileDownloadSize>10552428800</maxFileDownloadSize>
>>>>>>>           </NetcdfSubsetService>
>>>>>>>
>>>>>>> Then I realized that somehow, the file is being seen as much larger than
>>>>>>> 3.1G, which is an odd behavior...I shouldn't have to specify 10G to
>>>>>>> cover a 3.1G file.  So, I'll have to dig and see what's going on.
>>>>>>>
>>>>>>> Have a good weekend,
>>>>>>>           Lansing
>>>>>>>
>>>>>>> On 4/11/2014 5:31 PM, Luke Sheneman wrote:
>>>>>>>> New Client Reply: TDS java exception 
>>>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>>>
>>>>>>>> Thanks Lansing -
>>>>>>>>
>>>>>>>> I've attached the latest log files.
>>>>>>>>
>>>>>>>> I am somewhat familiar with the data.   These are downscaled climate
>>>>>>>> model outputs.  Some of the data are historical and some are
>>>>>>>> predictive.   The one you were looking at is predicting climate
>>>>>>>> scenarios for periods in the future (the years 2086 through 2100).   I
>>>>>>>> don't see anything weird with those dates in the data that you were
>>>>>>>> looking at...
>>>>>>>>
>>>>>>>> Have a great weekend, and I look forward to working with you soon to 
>>>>>>>> get
>>>>>>>> the java exception.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>           -Luke
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Luke Sheneman, Ph.D
>>>>>>>> Technology and Data Services Manager
>>>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>>>> http://www.northwestknowledge.net
>>>>>>>>
>>>>>>>> address@hidden
>>>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/11/14, 3:58 PM, Unidata THREDDS Support wrote:
>>>>>>>>> baby steps...
>>>>>>>>>
>>>>>>>>> When you get a moment, could you repeat the log capture procedure?  
>>>>>>>>> I'm about to head out for the day, but I'll pull this up again on 
>>>>>>>>> Monday.  I looked at one of the files already, which I downloaded 
>>>>>>>>> using the http server.  Are you familiar with the data, by chance?  I 
>>>>>>>>> notice that the calendar dates (Gregorian, days since 1901-01-01) are 
>>>>>>>>> for 2086-2100.  Is this right, or is something else fishy here?
>>>>>>>>>
>>>>>>>>> Have a good weekend!
>>>>>>>>>
>>>>>>>>> -Lansing
>>>>>>>>>
>>>>>>>>> Ticket Details
>>>>>>>>> ===================
>>>>>>>>> Ticket ID: SYD-491615
>>>>>>>>> Department: Support THREDDS
>>>>>>>>> Priority: Normal
>>>>>>>>> Status: Open
>>>>>>>>>
>>>>>>>> Ticket Details
>>>>>>>> ===================
>>>>>>>> Ticket ID: SYD-491615
>>>>>>>> Department: Support THREDDS
>>>>>>>> Priority: Normal
>>>>>>>> Status: Open
>>>>>>>> Link:  
>>>>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>>>>> Ticket Details
>>>>>>> ===================
>>>>>>> Ticket ID: SYD-491615
>>>>>>> Department: Support THREDDS
>>>>>>> Priority: High
>>>>>>> Status: Open
>>>>>>>
>>>>>> Ticket Details
>>>>>> ===================
>>>>>> Ticket ID: SYD-491615
>>>>>> Department: Support THREDDS
>>>>>> Priority: High
>>>>>> Status: Open
>>>>>> Link:  
>>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>>> Ticket Details
>>>>> ===================
>>>>> Ticket ID: SYD-491615
>>>>> Department: Support THREDDS
>>>>> Priority: High
>>>>> Status: Open
>>>>>
>>> Ticket Details
>>> ===================
>>> Ticket ID: SYD-491615
>>> Department: Support THREDDS
>>> Priority: High
>>> Status: Open
>>> Link:  
>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>
>>
>> Ticket Details
>> ===================
>> Ticket ID: SYD-491615
>> Department: Support THREDDS
>> Priority: High
>> Status: Open
>
>
> Ticket Details
> ===================
> Ticket ID: SYD-491615
> Department: Support THREDDS
> Priority: High
> Status: Open
> Link:  
> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770



Ticket Details
===================
Ticket ID: SYD-491615
Department: Support THREDDS
Priority: High
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.