Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Hi John, I did not know about the NcML Aggregation capability in the Java cdm library, thank you for pointing this out. So no, it was simply opening all the netcdf resources using a for-each loop over a Set<URI> using NetcdfFile.open( uri.toString() ). Yes, both HTTP servers support range requests and I can observe range requests and responses using Java network debug settings. One example visible to you would be the medium range members at https://nomads.ncep.noaa.gov/pub/data/nccf/com/nwm/prod/ (I would give more exact links, but the first subdirectory changes day-to-day). Below is an example to reproduce the issue. It requires either passing a list of resources that are larger than the default maximum heap on ones machine or setting the maximum default heap to be smaller than the size of the resources in bytes times ten million or the size of the resources themselves (whichever is smaller). I also attempted to attach the complete example in compressed form, with jars and all, but that was blocked. One can reconstruct it by looking at the commands below and finding the appropriate jars. Jesse -- Contractor, ERT, Inc. Federal Affiliation: NWC/OWP/NOAA/DOC package com.example; import java.io.IOException; import java.util.Set; import java.util.HashSet; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import ucar.nc2.NetcdfFile; public class CdmHttpResourceOomeExample { private static final Logger LOGGER = LoggerFactory.getLogger( CdmHttpResourceOomeExample.class ); /** * Opens each argument using NetcdfFile.open(), keeps all of them open, then * after opening them all, closes them all. * * @param args An array of URLs to open as netCDF resources. * @throws IOException When opening an arg as a netCDF resource fails. * @throws IllegalArgumentException When no arg is specified. */ public static void main( String[] args ) throws IOException { if ( args == null || args.length < 1 ) { throw new IllegalArgumentException( "Please specify a URL." ); } Set<NetcdfFile> ncResources = new HashSet<>(); try { for ( String arg : args ) { NetcdfFile ncResource = NetcdfFile.open( arg ); ncResources.add( ncResource ); } } finally { for ( NetcdfFile ncResource : ncResources ) { try { ncResource.close(); } catch ( IOException ioe ) { LOGGER.warn( "Failed to close {}", ncResource ); } } } } } To compile on Windows: %JAVA_HOME%\bin\javac -cp cdm-5.1.0.jar;slf4j-api-1.8.0-beta4.jar;. com\example\CdmHttpResourceOomeExample.java To run simplest example on Windows: %JAVA_HOME%\bin\java -Xms10m -Xmx10m -cp cdm-5.1.0.jar;slf4j-api-1.8.0-beta4.jar;logback-core-1.3.0-alpha4.jar;logback-classic-1.3.0-alpha4.jar;httpservices-5.1.0.jar;httpcore-4.4.12.jar;httpclient-4.5.10.jar;httpmime-4.5.10;jcl-over-slf4j-1.8.0-beta4.jar;re2j-1.3.jar;. com.example.CdmHttpResourceOomeExample https://any/nc/resource/bigger/than/ten/million/bytes.nc
netcdf-java archives: