[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: WDSSII Files

Kevin L. Manross wrote:


Hi John,
I have multiple issues that I'm struggling with while trying to modify a WDSSII netCDF file so that attempts to meet accepted netCDF standards. If you can bear with me, I'd like to lay them out and if you might have the time to answer them, I'd love to hear what you have to say.
1) I mentioned on the THREDDS listserv that we have historically used sparse grids. I'd be happy to try and work around this, but I don't think we can give up the compression we're getting from using the sparse grid to eliminate space for "missing values".
its possible that netcdf-4 compression will give you similar results, so lets try that first. It may take a few iterations to really check that out.
You mentioned trying netCDF-4 to try and get the compression back. I built netCDF-4.0.1 on my RHEL5 machine and simply tried running ncgen4 -k 3 to make a netCDF-4 file out of the CDL file I made by dumping an existing (non sparse grid) data file. I didn't get any compression out of this, but that's not too surprising, as the little that I've been able to glean suggests the the compression is in a C (or FORTRAN) method for the variable. Is that the route I need to go? 
yes, youll need to set deflate=true on each data variable, see:

http://www.unidata.ucar.edu/netcdf/docs_snapshot/netcdf-c.html#nc_005fdef_005fvar_005fdeflate

im not sure what deflate level to set, try a couple, presumable higher = 
smaller, slower.

youll also want to play with chunking parameters:

http://www.unidata.ucar.edu/software/netcdf/docs_snapshot/netcdf-c.html#nc_005fdef_005fvar_005fchunking

alhtough you could do that on pass 2.
2) Along the lines of saving space, can I set up my netCDF file without explicitly declaring my domain grid (i.e., lat/lon)? Perhaps it won't take up that much space, but we're dealing with a 3000x6200 (latxlon) grid and maybe the 9200 gridpoint values for the domain won't add much to the overall size.
is this a uniform grid? if so, one sets 1D lat and 1D lon and its a trivial sze.
3) Is there a way to write netCDF-4 files from the java libraries yet, or is the only way to write netCDF-4 from the tarball that needs to be compiled?
not yet, were working on a JNI interface, but its not ready; youll need the netcdf-4 C library anyway. I assume you already have a C/Fortran program that writes the files? If so, tweaking that will be the fastest thing to do.
4) When I try to open up one of our non sparse grid netCDF files in the toolsUI application, I get an error that says "New variable (cref) must have a datatype attribute". This only occurs when using toolsUI 4.x. When I use toolsUI 2.2x, it seems to not have a problem
can you send me a sample file with this problem? which tab in ToolsUI are you using?
5) To try to set up my existing netCDF file (non sparse grid) to be CDM compliant, it seems that I mainle need to improve on the header information. (referencing the sample file I submitted to the THREDDS listserv) Does that seem correct? If so, should I just try some adjustments using the NcML capabilities in the toolsUI app and write out the file and then test the CDM compliance?
yes, assuming the compression makes the files small enough, adding the CF metadata is not too hard.
I know I'm throwing a bunch at you. I've got some support to get our data up to standards, and I'd really like to try to achieve this. If you can help out in any way, I'd be very grateful! Thanks! 
would be great to have your data in THREDDS!