Tony, I'm traveling this week but I'll try to get back to you abut this stuff... James On Sep 26, 2011, at 10:11 AM, Tony Stocker wrote: > James, > > Okay, we've tested this and everything works fine now. I do have two > quick questions: > > 1. Can you describe what we're doing "wrong" with our metadata? > I mean we're obviously doing something that you're not expecting > if you're having to maintain a code block with a comment with our > name in it. I can't promise anything, but I'd at least like to > pass this along to the scientists and algorithm developers to see > if perhaps they could agree on making changes to make the metadata > more "friendly". > > 2. I tried to install the latest HDF5 handler while doing this > 1.7.1 rpm install, but unfortunately the latest version of the > HDF 5 rpms is now 1.8.7 and your handler wants 1.8.5 only. So my > question is, is there any way to make your handler rpms accept an > => (equal to or greater) with regards to the underlying library > dependency version? Or do things absolutely have to be fixed to a > particular rpm version? > > Thanks again for all the help! > > Tony > > > On Tue, 6 Sep 2011, Gallagher James wrote: > >> >> On Sep 6, 2011, at 6:13 AM, Tony Stocker wrote: >> >>> James, >>> >>> Is there an updated package that we can install and use to see if the >>> original problem is fixed? >> >> Yes - The Hyrax 1.7.1 has a new package for the HDF4 handler that should >> work as a drop in replacement. I did not test that for both the 32 and >> 64-bit linux builds, but it's likely to work. In the worst case, the changes >> outside of the hdf4 handler and the BES (but not a change to the BES's ABI). >> >> Here's a link to the release: http://www.opendap.org/download/hyrax/1.7.1 >> >> Here's a link to the 64-bit Linux RPM for the hdf4 handler: >> http://www.opendap.org/pub/binary/hdf4_handler/centos5.2/3.9.3_x86_64 >> >> James >> >>> >>> Tony >>> >>> On Wed, 17 Aug 2011, Gallagher James wrote: >>> >>>> >>>> On Aug 17, 2011, at 2:41 PM, Gallagher James wrote: >>>> >>>>> >>>>> On Aug 17, 2011, at 12:50 PM, Tony Stocker wrote: >>>>> >>>>>> >>>>>> James, >>>>>> >>>>>> I've sent your write-up along to our algorithm developers. I will say >>>>>> that this issue is present in all of our HDF files in version 7, >>>>>> regardless of algorithm. Beyond that I'm afraid that as systems guy, >>>>>> I'm not in much of position to comment on the approach. >>>>> >>>>> I'm looking at the hdf4 handler and there are number of issues there. One >>>>> is that the handler seems to think that every dimension of every array >>>>> must have a name, so those without a name in the data file get names >>>>> 'fakeDim_n' where n is 0, 1, 2, ... I'd like to silence that and also >>>>> remove the attributes that are also being created for those 'fake dims' >>>>> because it looks like those are only the 'fake dim' names and nothing >>>>> more. It's easy to use the API to read the names (or discover there's no >>>>> name associated with a particular dimension) from libdap or any one of >>>>> the other APIs. >>>> >>>> Gads, I just found this comment in the handler code: // Reduce the fakeDim >>>> list. FakeDim is found to be used by TRMM team. >>>> >>>> Comments? >>>> >>>> PS. I have fixed the bug that prompted this email w/o resorting to >>>> removing the fakeDims or their 'name' attributes. I'd still like to remove >>>> the latter if possible (that is, remove the fakeDim attribute 'String name >>>> "fakeDim0"') but will wait on what you say. >>>> >>>>> >>>>> Do you (and by extension NASA, THG) see any problem with my making the >>>>> handler behave a little differently so long as we can get back the old >>>>> behavior (I'll use a run-time parameter to switch on the new behavior)? >>>>> It may be that the new behavior is required to read some files. I'm not >>>>> sure about that last part, I have a fair amount more work to do on this >>>>> problem. >>>>> >>>>> James >>>>>> >>>>>> Tony >>>>>> >>>>>>> On Tue, Aug 16, 2011 at 6:31 PM, Gallagher James <address@hidden> wrote: >>>>>>>> >>>>>>>> On Aug 11, 2011, at 1:30 PM, Tony Stocker wrote: >>>>>>>> >>>>>>>>> James, >>>>>>>>> >>>>>>>>> Any word on this issue? >>>>>>>> >>>>>>>> Tony, >>>>>>>> >>>>>>>> Are the VData variables (I'm not sure if 'variables' is the right >>>>>>>> word) empty? When I look at the two kinds of files, in the v7 file >>>>>>>> there are a number of VData that, when viewed with vshow, look like: >>>>>>>> >>>>>>>> vs:3 <1962/10340> nv=0 i=0 fld [SDS variable] vsize=4 (NoName {SDSVar}) >>>>>>>> >>>>>>>> That is the field is called 'SDS variable', the name is NoName and nv >>>>>>>> is 0. In the v6 files (the ones that work) there are no entries with >>>>>>>> nv=0; all of the nv values are >= 1 in the files that work. >>>>>>>> >>>>>>>> In the documentation for VSsetfield() (which must be called before a >>>>>>>> Vread()) it says that if the VData is empty (I'm paraphrasing here) >>>>>>>> then the function will return -1 (the code for failure). However, >>>>>>>> prior to calling that function we are testing a number of other >>>>>>>> things. These VData have all of the following: a fieldname, >>>>>>>> fieldorder, fieldsize, and fieldtype. >>>>>>>> >>>>>>>> I will try building the handler so that it treats this particular >>>>>>>> error as meaning the vdata is empty, but it's very hard for me to tell >>>>>>>> what's what in these very large HDF4 files. While I work on that, can >>>>>>>> you let me know if this seems to be a promising tack? >>>>>>>> >>>>>>>> HDF folks: Can you weigh in on this? Am I close to understanding the >>>>>>>> semantics of VSsetfield()? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> James >>>>>>>> >>>>>>>> PS. The ticket is http://scm.opendap.org:8090/trac/ticket/1793 >>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Tony >>>>>>>>> >>>>>>>>> On Thu, 21 Jul 2011, address@hidden wrote: >>>>>>>>> >>>>>>>>>> Tony, >>>>>>>>>> I'm out for the next few days, but do send those files. The problem >>>>>>>>>> is in the hdf4 handler. Most likely anyway. >>>>>>>>>> James >>>>>>>>>> Sent from my Verizon Wireless Phone >>>>>>>>>> ----- Reply message ----- >>>>>>>>>> From: "Tony Stocker" <address@hidden> >>>>>>>>>> Date: Thu, Jul 21, 2011 2:40 pm >>>>>>>>>> Subject: [support] Re: Could not read vdata records >>>>>>>>>> To: "OPENDAP support" <address@hidden> >>>>>>>>>> Cc: "Owen Kelley" <address@hidden> >>>>>>>>>> And now to actually include the attachments (*sigh*) >>>>>>>>>> On Thu, 21 Jul 2011, Tony Stocker wrote: >>>>>>>>>>> Hi Guys, >>>>>>>>>>> >>>>>>>>>>> Okay, quick and dirty description of the problem. We have started >>>>>>>>>>> reprocessing our science data, i.e. developers have new algorithms >>>>>>>>>>> and so we >>>>>>>>>>> re-run everything. Our old data is V6 and the new data is V7. We >>>>>>>>>>> keep both >>>>>>>>>>> around for comparison purposes. >>>>>>>>>>> >>>>>>>>>>> OpenDAP has been working fine with our V6 data, and continues to do >>>>>>>>>>> so. >>>>>>>>>>> >>>>>>>>>>> However it is choking on all of our V7 data, producing the error >>>>>>>>>>> message: >>>>>>>>>>> "libdap exception building response: error_code = 1001: Could not >>>>>>>>>>> read Vdata >>>>>>>>>>> records." >>>>>>>>>>> >>>>>>>>>>> I'm going attaching two text files containing debug output of a >>>>>>>>>>> successful V6 >>>>>>>>>>> open and an unsuccessful V7 open of the same day/orbit: >>>>>>>>>>> od.v6.19980101.1c21.00539-debug.txt i.e. WORKING FILE >>>>>>>>>>> od.v7.19980101.1c21.00539-debug.txt i.e. NOT working FILE >>>>>>>>>>> >>>>>>>>>>> I also wanted to see if you guys would let me send you both of >>>>>>>>>>> these HDF >>>>>>>>>>> files (the v6 and v7 ones) to see if you guys see the same behavior >>>>>>>>>>> on your >>>>>>>>>>> installations. I haven't attached these to this email however >>>>>>>>>>> since each is >>>>>>>>>>> ~150 MB in size. I think it would be helpful to see if the problem >>>>>>>>>>> is >>>>>>>>>>> universal or localized to our installation. Let me know if you're >>>>>>>>>>> willing to >>>>>>>>>>> take and test the files, and how you'd like to get them (email, >>>>>>>>>>> ftp, etc.) >>>>>>>>>>> >>>>>>>>>>> I also want to mention that we have verified that the V7 HDF files >>>>>>>>>>> are not >>>>>>>>>>> corrupt or otherwise unreadable in themselves by using two other HDF >>>>>>>>>>> viewer/reader programs on them, and they come up fine, our only >>>>>>>>>>> problem seems >>>>>>>>>>> to be with the OpenDAP server reading them. >>>>>>>>>>> >>>>>>>>>>> Here are our software product versions: >>>>>>>>>>> bes-3.9.0-1 >>>>>>>>>>> bes-devel-3.9.0-1 >>>>>>>>>>> dap-server-4.1.0-1 >>>>>>>>>>> libdap-3.11.0-1 >>>>>>>>>>> libdap-devel-3.11.0-1 >>>>>>>>>>> olfs-1.7.1 >>>>>>>>>>> freeform_handler-3.8.2-1 >>>>>>>>>>> hdf4_handler-3.9.1-1 >>>>>>>>>>> hdf5_handler-1.4.3-1 >>>>>>>>>>> netcdf_handler-3.9.2-1 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Look forward to hearing back from you guys, since this is >>>>>>>>>>> perplexing and >>>>>>>>>>> pretty much out of our balliwick. >>>>>>>>>>> >>>>>>>>>>> Thanks! >>>>>>>>>>> >>>>>>>>>>> Tony >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> +-----------------------------------+ >>>>>>>>>> | Tony Stocker | >>>>>>>>>> | Systems Programmer | >>>>>>>>>> | TSDIS/TRMM Code 610.2 | >>>>>>>>>> | 301-614-5738 (office) | >>>>>>>>>> | 301-614-5269 (fax) | >>>>>>>>>> | address@hidden | >>>>>>>>>> +-----------------------------------+ >>>>>>>>>> -- >>>>>>>>>> This message has been scanned for viruses and >>>>>>>>>> dangerous content by MailScanner, and is >>>>>>>>>> believed to be clean. >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> +-----------------------------------+ >>>>>>>>> | Tony Stocker | >>>>>>>>> | Systems Programmer | >>>>>>>>> | TSDIS/TRMM Code 610.2 | >>>>>>>>> | 301-614-5738 (office) | >>>>>>>>> | 301-614-5269 (fax) | >>>>>>>>> | address@hidden | >>>>>>>>> +-----------------------------------+ >>>>>>>>> -- >>>>>>>>> This message has been scanned for viruses and >>>>>>>>> dangerous content by MailScanner, and is >>>>>>>>> believed to be clean. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> James Gallagher >>>>>>>> jgallagher at opendap.org >>>>>>>> 406.723.8663 >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> +-----------------------------------+ >>>>>> | Tony Stocker | >>>>>> | Systems Programmer | >>>>>> | TSDIS/TRMM Code 610.2 | >>>>>> | 301-614-5738 (office) | >>>>>> | 301-614-5269 (fax) | >>>>>> | address@hidden | >>>>>> +-----------------------------------+ >>>>>> -- >>>>>> This message has been scanned for viruses and >>>>>> dangerous content by MailScanner, and is >>>>>> believed to be clean. >>>>>> >>>>> >>>>> -- >>>>> James Gallagher >>>>> jgallagher at opendap.org >>>>> 406.723.8663 >>>>> >>>> >>>> -- >>>> James Gallagher >>>> jgallagher at opendap.org >>>> 406.723.8663 >>>> >>>> >>> >>> -- >>> +-----------------------------------+ >>> | Tony Stocker | >>> | Systems Programmer | >>> | TSDIS/TRMM Code 610.2 | >>> | 301-614-5738 (office) | >>> | 301-614-5269 (fax) | >>> | address@hidden | >>> +-----------------------------------+ >>> >>> -- >>> This message has been scanned for viruses and >>> dangerous content by MailScanner, and is >>> believed to be clean. >>> >> >> -- >> James Gallagher >> jgallagher at opendap.org >> 406.723.8663 >> >> > > -- > +-----------------------------------+ > | Tony Stocker | > | Systems Programmer | > | TSDIS/TRMM Code 610.2 | > | 301-614-5738 (office) | > | 301-614-5269 (fax) | > | address@hidden | > +-----------------------------------+ > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > -- James Gallagher jgallagher at opendap.org 406.723.8663
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.