Re: [netcdf-java] OutOfMemoryError when opening an ensemble forecast dataset (many netcdf resources) via HTTP

  • To: John Caron <jcaron1129@xxxxxxxxx>
  • Subject: Re: [netcdf-java] OutOfMemoryError when opening an ensemble forecast dataset (many netcdf resources) via HTTP
  • From: Jesse Bickel - NOAA Affiliate <jesse.bickel@xxxxxxxx>
  • Date: Wed, 23 Oct 2019 12:14:42 -0500
Hi John,

I did not know about the NcML Aggregation capability in the Java cdm
library, thank you for pointing this out. So no, it was simply opening
all the netcdf resources using a for-each loop over a Set<URI> using uri.toString() ).

Yes, both HTTP servers support range requests and I can observe range
requests and responses using Java network debug settings. One example
visible to you would be the medium range members at (I would give
more exact links, but the first subdirectory changes day-to-day).

Below is an example to reproduce the issue. It requires either passing
a list of resources that are larger than the default maximum heap on
ones machine or setting the maximum default heap to be smaller than
the size of the resources in bytes times ten million or the size of
the resources themselves (whichever is smaller). I also attempted to
attach the complete example in compressed form, with jars and all, but
that was blocked. One can reconstruct it by looking at the commands
below and finding the appropriate jars.


Contractor, ERT, Inc.
Federal Affiliation: NWC/OWP/NOAA/DOC

package com.example;

import java.util.Set;
import java.util.HashSet;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import ucar.nc2.NetcdfFile;

public class CdmHttpResourceOomeExample
    private static final Logger LOGGER = LoggerFactory.getLogger(
CdmHttpResourceOomeExample.class );

     * Opens each argument using, keeps all of them open, then
     * after opening them all, closes them all.
     * @param args An array of URLs to open as netCDF resources.
     * @throws IOException When opening an arg as a netCDF resource fails.
     * @throws IllegalArgumentException When no arg is specified.

    public static void main( String[] args ) throws IOException
        if ( args == null || args.length < 1 )
            throw new IllegalArgumentException( "Please specify a URL." );

        Set<NetcdfFile> ncResources = new HashSet<>();

            for ( String arg : args )
                NetcdfFile ncResource = arg );
                ncResources.add( ncResource );
            for ( NetcdfFile ncResource : ncResources )
                catch ( IOException ioe )
                    LOGGER.warn( "Failed to close {}", ncResource );

To compile on Windows:
   %JAVA_HOME%\bin\javac -cp cdm-5.1.0.jar;slf4j-api-1.8.0-beta4.jar;.

To run simplest example on Windows:
    %JAVA_HOME%\bin\java -Xms10m -Xmx10m -cp

  • 2019 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: