Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Hi John, Yes, it is technically feasible to open a portion of a forecast and close that portion of the forecast, iterating one by one over hundreds of timesteps, meanwhile collecting the data in that forecast. However, one forecast spans many netCDF resources, and many of those forecasts are included in the same netCDF resources. I agree that the use of try-with-resources and Closeable/AutoCloseable are more desirable in general. The abstraction I am building implements Closeable and manages these several hundred netCDF resources. All it does is read one of those forecasts when asked to. It could do this in serial or in parallel. This same abstraction works for both HTTP resources and filesystem resources thanks to the good abstraction in the Java cdm library. The question to me is this: why does the OutOfMemoryError occur only with HTTP resources but not with filesystem resources? Why doesn't the filesystem version allocate a ten million byte buffer on opening the file while the HTTP version does? I have observed that reducing the buffer size removes the OutOfMemoryError but involves some tradeoffs. Therefore I speculate that some past tuning for a particular use case caused the ten million byte buffer. It is neither here nor there how or why, I was just curious. One can observe this by passing a 13MiB netCDF local file to the same program: no OutOfMemoryError. Pass the same 13MiB netCDF resource but via HTTP and one gets an OutOfMemoryError. As far as using the Aggregation capability, glancing over the software it looks like this might automatically read all the bytes of at least several netCDF resources and then write them to temporary storage on the filesystem. If true, this is contrary to the goal. I am trying to avoid transferring multiple GiB over a network when the software is only interested in KiB or MiB. The HTTPRandomAccessFile seems to do the right thing by using Range requests. The only strange thing is the allocation of such a large amount of resources to only open a netCDF resource. If I am wrong about the Aggregation abstraction (i.e. it does not auto-download to a filesystem) then I would be interested to try that instead. In any case, it appears the issue is resolved with the merge of https://github.com/Unidata/netcdf-java/pull/138 which makes the property available to whatever launches the JVM. Thank you so much for taking the time to reply and suggest alternatives. Thanks for the netCDF Java library! Jesse -- Contractor, ERT, Inc. Federal Affiliation: NWC/OWP/NOAA/DOC
netcdf-java archives: