Java, netcdf files and IDV consumption

Richard P Signell wrote:
Is there a place on the web where we can find examples of 1D, 2D, and 3D netcdf files that *do* work with IDV?


Hi Richard, I saw your posting. I am currently finishing a converter from mm5v3 model output files to CF-1.0 compliant .nc files. A win is direct consumption by IDV, release 1.2 at least. So far it has been working ;)

If you load IDV 1.2 and go to Data Source Chooser|URLs and type

http://metoc.apl.washington.edu/uwme/2005090100-eta-d1.nc

then IDV should download a netcdf file from our server which includes 2D and 3D model output fields.

If you wanted to see the .nc format, just type the same URL into a regular browser and Save As to your local system and browse with ncdump.

The netcdf functionality of our server is somewhat experimental at the moment, since I'm adding handling of the cross and dot staggering of mm5 grids, but at any time, the 'file' you get should be viewable in IDV.

I learnt most if not all about the CF conventions from the CF web page, and from looking at the RUC.cdl example at ucar, and from answers from Don Murray!

Also, I have learned that due to the way netcdf files are constructed, IDV can actually ask for sections of data without needing all of the file contents. Using the httpclient from Apache commons, IDV hits the above url and in the http request says it wants bytes in the range say 0-20000. So our web server has to understand this (it's standard http 1.1, we use Tomcat 5.5.9). Think of this first data chunk as the table of contents of a book, with page numbers to various chapters/sections. Then when the IDV user wants to look at some field, IDV works out which chapter (byte offset in fact) it's in from the TOC, and asks for a new range. This is neat, since our mm5/nc files are ~50MB which could take a while to transfer, while a single field/time combo may only be ~100K.

You can see all these range requests by (a) looking in the web server access logs for 206 (partial content) rather than 200 (OK) records, and by (b) using tcpmon (google it) as a middle man to view the http messages back and forth. That's an invaluable tool I use it all the time for http debugging. tcpmon becomes the client of the web site and the server for IDV, forwarding bytes in both directions while also displaying them in a swing panel set.

I will try to build a catalog (adde??) for our data set, at the moment you have to know the url format, which is YYYYMMDDHH-model-nest.nc

where HH is 00 or 12, model can be eta, gfs (plus some others), domain is "d1" or "d2". The data goes back to about 2001. See our web site at http://metoc.apl.washington.edu/uwme/ for more details on what data we are processing/offering up.


BTW if anyone is interested from a Java web app view, I use a servlet filter technique to launch the mm5 to nc conversion at request time, and session expiry to clear up local .nc files. Our mm5 data set is 3TB and we don't have disk space for a local .nc copy as well, so we generate just what users want ;)

I hope this rambling is all of use to the community, I had been meaning to 'advertise' our .nc data anyways, so your post struck a chord. Feel free to ask any questions...

Stuart