Re: netcdf 3, java, combining with java.util.zip

Ken Pavelle wrote:

I was using the ncCore jarfile. I switched to the netcdf2-2.jar and it works. Unfortunately, this leads to a bigger issue. My goal in switching from netcdf2.1 to 2.2 is to add the ability to read zipped netcdf files in my web applet. The thought was that reading 70-500K zipped files as compared to 220-2800K uncompressed files would significantly reduce user bandwith requirements. However, I've had to shoot myself in the foot with the size of the new jarfiles:

   netcdf_2.1.jar + HTTPClient.jar = 220368 + 166656 ~ 387K
netcdf_2.2.jar + commons-httpclient-2.0.1.jar + commons-logging.jar = 1499643 +224754 + 38015 ~ 1762K

Am I throwing the baby out with the bathwater here? Is there a way to pick and choose components of the full implementation plus the commons jars, and create a netcdf jar that still meets my needs but isn't so large?

Hi Ken:

The best thing to do is to compile just those parts of the library that you need. This is a bit tricky though.

If you were getting away with just using the core, then i'm guessing that you need a pretty minimal set. Maybe you can just use the core + HTTP access ?? Actually we can use the built-in java HTTP access, and you dont even need the commons jars.

Do you know how to build from nj22 source? How about rearranging the contents of jars?


John Caron wrote:

Ken Pavelle wrote:

John,

Sorry to keep bugging you, but I'm (a) determined to get this work, and (b) frustrated after 3 days of staring at the code w/o making progress.

Here's the relevant portions of the code that works with the jar from netcdf 2.1:
           URL ncdURL = new URL ("http", thishost, TNCFile.GetName ());
           NetcdfFile ncdf = new NetcdfFile (ncdURL);
           pvar = ncdf.findVariable ("amountofprecip");

My first goal is to get the code to work with the new API. The following code:
           URL ncdURL = new URL ("http", thishost, TNCFile.GetName ());
           NetcdfFile ncdf = NetcdfFile.open (ncdURL.toString());
           pvar = ncdf.findVariable ("amountofprecip");
yields the following exception:
java.lang.NoClassDefFoundError: ucar/unidata/io/http/HTTPRandomAccessFile3
Any thoughts?  Is this a problem between ucar.nc2 and ucar.unidata.io?



make sure HTTPRandomAccessFile3 is in netcdf2-2.jar and that jar is in your classpath.

looks like you are using ncCore.jar instead of netcdf.jar. The latter is needed to do HTTP reads.


My ultimate goal is to use NetcdfFile.openInMemory. I can open a GZIPInputStream and read the contents into a byte[] memory buffer. My question is, what is the role of the first argument (String location)? Will the following line work the way I hope? NetcdfFile ncdf = NetcdfFile.openInMemory (ncdURL.toString(), buffer);



the location is only used as the name of the dataset. You can actually make it anything you want.


Thanks again for any additional help you can provide.

Regards,
Ken Pavelle
Hydrologist / Webmaster
NOAA / National Weather Service
Arkansas-Red Basin River Forecast Center


John Caron wrote:

Ken Pavelle wrote:

John,

I'm having two problems upgrading my applet to the new jar file. (Same results for ncCore-2.0.06 and 2.0.09). First, I get the following runtime exception at the line where I set up my IndexIterator:

Exception in thread "AWT-EventQueue-2" java.lang.IncompatibleClassChangeError
    at RFCPanel.paintComponent(RFCPanel.java:691)






Any thoughts? Does this imply I don't have my array defined properly? Has something changed with ucar.ma2.Array, ucar.ma2.IndexIterator, ucar.nc2.NetcdfFile or ucar.nc2.Variable? Note, the same code works when I use netcdf.jar version 2.1beta.





there were some changes in 2.2.09. It sounds like you should recompile all your code against the new library. You may see some minor API changes.


Second, my main goal in upgrading to 2.2 is to be able to read .nc.gz files. My original idea was to open the file with a GZIPInputStream, read the data into a byte[] array, and then use NetcdfFile.openInMemory. Looking through the javadoc for 2.2, I get the impression that NetcdfFile.open takes the .gz extension into account, meaning I don't have to go through the GZIPInputStream to buffer to OpenInMemory. Is my impression correct?





NetcdfFile.open will look for compressed files like gz, decompress them onto disk, and then open the decompressed file. SInce you are trying to do everything in memory, it looks like you will need to keep doing that yourself.


One final question. Any idea when you'll be ready to launch a beta version of 2.2? I'm hearing concerns about putting code into production that is based on alpha versions.





A valid concern. Its because of the the minor API tweaks that we are still calling it alpha. The tweak was needed to improve performance to an acceptable level for a certain special case. Otherwise the code is getting pretty good testing, and there are several production packages using it, including the IDV. Typically whatever changes made are minor, and a recompile will discover and quickly fix them.

By the end of the month we'll probably go to beta, although there may be some sections of the library that will remain alpha code, maybe the "higher level" data types and the UI.

Thanks for your patience.


Thanks (again) for your time and help.  Regards,
Ken Pavelle


John Caron wrote:

Ken Pavelle wrote:

John,

Can you help point me in the right direction? We have only one novice Java programmer here, and I'm not even sure how big my project is.

We have a Java applet that reads netCDF files and interactively displays the data ( http://www.srh.noaa.gov/abrfc/precip/zoom.php ). I want to save user bandwith by reading gzipped netcdf files instead. I can read the zipped file using java.util.zip, but the netCDF Java library only reads from netCDF files, not from a memory buffer.

How hard will it be for a decent Java programmer to extend the netCDF java library (specifically NetcdfFile?) to read from buffer instead of file?

Thank you for your time. (....even moreso if you have the time to reply!)

Regards,
Ken Pavelle
NOAA, National Weather Service







Hi Ken:

unzip the file into a byte array, then call

 NetcdfFile.openInMemory(String location, byte[] data)

then it should work as usual

make sure you have the latest version (2.2.06)