[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [netcdf-java] Differing results reading an unlimited dimension between Java and C (?) binary distribution



Hi Jean-Michel.:

Ok, heres the situation:

Its actually illegal to use STRING here:

   writeableFile.addVariable("cellId", DataType.STRING, spatialDim);

you should either use:

   writeableFile.addVariable("cellId", DataType.CHAR, spatialDim);

or

   writeableFile.addStringVariable("cellId",  spatialDim, max_strlen);

However, the library did not catch this problem, so it put an incorrect "vsize" into the netcdf header.

The java-library uses this vsize, even though incorrect, and so everything works ok, but the C library ignores it, and so reads bad values.

Im not sure whether the cellId variable values are correct, however.

So technically you have invalid files, which you should rewrite if you need them to interoperate with C programs. I will release a new version of the library tomorrow which should prevent the previous error.

Let me know if I can clarify anything else for you. Sorry for the problems.

John



address@hidden wrote:
Hi John,

I have put a small data set with the code that produce it, and exhibiting the 
behavior mentioned, in ftp://ftp.csiro.au/netCDFJava/.
The API version is
## Version 4.0.44.20090527.1935
## Build Date = 2009-05-27 19:35:38

Of course let me know if you need further information/clarifications.
Regards

-----Original Message-----
From: John Caron [mailto:address@hidden] Sent: Tuesday, 2 June 2009 10:07 PM
To: Perraud, Jean-Michel (CLW, Black Mountain)
Cc: address@hidden
Subject: Re: [netcdf-java] Differing results reading an unlimited dimension 
between Java and C (?) binary distribution

Hi Jean-Michel:

Thats pretty strange. Can you put the file on an ftp or http server so we can 
look at it?

BTW, which version on netcdf-java are you using?


address@hidden wrote:
Hi,

I have a lat/lon/time data set for which the lat dimension is unlimited. On 
disk the file is ~14 GiB. It was created using the Java API. More precisely, a 
C# application calling netCDF-Java through IKVM; worked beautifully. Colleagues 
and I tried to read this file from other tools (R via Rnetcdf, ncdump, etc.). 
The values of the lat variable are not read correctly (either all zeroes, or a 
mix with other numbers presumably from other variables).

Has anyone had similar issues? This is really not a blocking issue and can live 
with it, but I am intrigued. Below are sample code and CDL. I cannot 
investigate more in-depth due to approaching milestones, but happy to provide 
clarifications if needed.
I am reluctant to cross-post for now, but this may well be an issue outside of 
the Java API.

Cheers

### From IronPython or Jython, using the Java API, read works as expected:
clr.AddReferenceToFileAndPath(r"C:\lib\netcdf\netcdfAll-4.0.dll")
clr.AddReferenceToFileAndPath(r"C:\bin\ikvm\bin\IKVM.OpenJDK.ClassLibrary.dll")
from ucar.nc2 import NetcdfFile
netcdfFile = NetcdfFile.open(r"\\somename\tmp\scenarioA.nc")
latVar = netcdfFile.findVariable("lat")
latArray = latVar.read()
latArray
# prints:
# <ucar.ma2.ArrayFloat+D1 object at 0x000000000000002B [-39.097 -39.047 -38.997 
-3
# 8.947 -38.897 -38.847 -38.797 -38.747 -38.697 -38.647 -38.597 -38.547 -38.497 
-3
netcdfFile.close()

### using ncdump from a linux distro on a ncks subset. Same behavior without 
the full 14GB data set.

netcdf smallScenarioA {
dimensions:
        lat = UNLIMITED ; // (4 currently)
        lon = 9 ;
        time = 40907 ;
variables:
        char cellId(lat, lon) ;
                cellId:long_name = "Cell ID" ;
        float lat(lat) ;
                lat:units = "degrees_north" ;
                lat:axis = "Y" ;
                lat:standard_name = "latitude" ;
        float lon(lon) ;
                lon:units = "degrees_east" ;
                lon:axis = "X" ;
                lon:standard_name = "longitude" ;
        float runoff(lat, lon, time) ;
                runoff:long_name = "runoff" ;
                runoff:units = "mm" ;
                runoff:missing_value = -9999.f ;
                runoff:_FillValue = -9999.f ;
        int time(time) ;
                time:units = "days since 1895-01-01" ;
                time:axis = "T" ;
                time:standard_name = "time" ;
                time:long_name = "time" ;

// global attributes:
                :history = "Sat May 30 19:58:55 2009: ncks -d lat,30,33 -d 
lon,160,168 scenarioA.nc smallScenarioA.nc" ;
data:

 cellId =
  "",
  "",
  "",
  "" ;

 lat = 0, 0, 0, 0 ;

 lon = 146.6, 146.65, 146.7, 146.75, 146.8, 146.85, 146.9, 146.95, 147 ;
_______________________________________________
netcdf-java mailing list
address@hidden
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/