[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #LYK-546339]: Problem with netCDF-Java when also using libjhdf5

personally, I am skeptical that having the same method in two
different libraries would cause any problem because I assume
that the controlling prototype object (e.g. hdf5 vs jhdf5)
is used to decide where to look.
I found this at atackoverflow where it discusses using different
classloaders.
https://stackoverflow.com/questions/3678421/is-it-possible-to-load-different-versions-of-the-same-dll-in-java

In any case, I think we need to look at the errors that you are seeing.
Can you provide some more information about that? 

> Hm. An idea, basing off of item 2:
> 
> It seems to me that the only way that this can be going wrong is that 
> libhdf5.so and libjhdf5.so have C functions that are identical in name (and 
> also in signature, AFAICT). So for example, In the JHDF5 code, when I call 
> H5.H5Aget_space, this calls native method _H5Aget_space, which calls a C++ 
> interface function called something like hdf_hdf5lib_H5__1H5Aget_space, which 
> in turn calls the H5Aget_space HDF5 function. The H5Aget_space function is 
> present in both libhdf5.so and libjhdf5.so.
> 
> If this is true, then it should be the case that renaming the H5Aget_space 
> function in JHDF5 to, say, H5Aget_space_j would prevent the unwanted call to 
> the wrong function (if indeed that is what is going on).
> 
> -PT
> 
> ?On 4/10/19, 12:51 PM, "Unidata netCDF Java Support" <address@hidden> wrote:
> 
> Some other thoughs:
> 1. It is possible also that just loading netcdf library is not
> sufficient to get HDF5 library loaded. You might need to do some
> explicit create/close or open/close thru the netcdf-c library to
> cause libhdf5 to be actually loaded.
> 2. Having two copies of the hdf5 library might also cause problems.
> you might need to modify the javahdf5 code to explicitly
> use the exact same library (not a copy or symbolic link)
> as netcdf library is using
> 
> 
> > Another problem may be that netcdf uses JNA, while apparently javaHDF5
> > uses JNI directly. But there is no way to fix that.
> >
> > > Order of loading: I have tried both orders. At first I thought netCDF 
> > > first was the problem, so I forced jarhdf5 to load first. However, in the 
> > > last 24 hours I've started seeing the problem when jarhdf5 is loaded 
> > > first as well.
> > >
> > > HDF5 version: to my knowledge they are both 1.8.19, but this cannot be 
> > > determined with certainty. The version used with netCDF is definitely 
> > > 1.8.19, but I cannot find any version information for the files that are 
> > > part of the jhdf5 package other than that the online documentation claims 
> > > that it is 1.8.19.
> > >
> > > -PT
> > >
> > > ?On 4/10/19, 11:56 AM, "Unidata netCDF Java Support" <address@hidden> 
> > > wrote:
> > >
> > > Unfortunately, we (Unidata) have not seen this issue either.
> > > At the Java level, having two different JNI APIs to the same
> > > library should probably be ok. However I might speculate
> > > about some possible issues:
> > > 1. order of loading - do you know if the netcdf-c library is being
> > > loaded before or after  jarhdf5-3.3.2.jar?
> > > 2. do you know if the HDF5 versions being used are the same?
> > >
> > > >
> > > > My project, the TESS Science Processing Operations Center (SPOC) at 
> > > > NASA Ames, is using the netCDF Java API to run our datastore: we use 
> > > > netcdf4-4-6.10.jar and netcdfAll-4-6.8.jar . As I understand it, the 
> > > > Java API then uses libnetcdf.so, which in turn makes use of libhdf5.so, 
> > > > both through JNA. We are using libhdf5.so.10.3.0 and libnetcdf.so.13.0 .
> > > >
> > > > So far so good, this has worked well for us for a couple of years. 
> > > > However:
> > > >
> > > > Recently we decided that we wanted to also use straight HDF5 in our 
> > > > Java code for another purpose (very long story). For this we are using 
> > > > the HDF5-Java implementation available from the HDF5 group: 
> > > > jarhdf5-3.3.2.jar, which in turn uses libjhdf5.so.3.3.2 . This library 
> > > > is accessed via JNI.
> > > >
> > > > Unfortunately, we observe problems when we use both of these libraries 
> > > > in a single executable on linux:
> > > >
> > > > What we observe is that there appear to be confusion problems when both 
> > > > libraries are loaded. So for example, the netCDF library loads and 
> > > > writes an array to disk; after that, we are unable to read HDF5 arrays 
> > > > and get errors when attempting to read attributes. As far as we can 
> > > > tell the HDF5 array in question is valid.
> > > >
> > > > I realize that this may be kind of vague. I’m writing in hopes that 
> > > > somebody at netCDF has heard about this kind of problem before and 
> > > > knows what to do! AFAICT the folks at HDF5 are not familiar with this 
> > > > issue.
> > > >
> > > > Thanks,
> > > > -PT
> > > >
> > >
> > > =Dennis Heimbigner
> > > Unidata
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: LYK-546339
> > > Department: Support netCDF Java
> > > Priority: Normal
> > > Status: Open
> > > ===================
> > > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > > Unidata inquiry tracking system and then made publicly available through 
> > > the web.  If you do not want to have your interactions made available in 
> > > this way, you must let us know in each email you send to us.
> > >
> > >
> > >
> > >
> > >
> >
> > =Dennis Heimbigner
> > Unidata
> >
> 
> =Dennis Heimbigner
> Unidata
> 
> 
> Ticket Details
> ===================
> Ticket ID: LYK-546339
> Department: Support netCDF Java
> Priority: Normal
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: LYK-546339
Department: Support netCDF Java
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.