[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [support] Re: Could not read vdata records

Tony,

I'm traveling this week but I'll try to get back to you abut this stuff...

James

On Sep 26, 2011, at 10:11 AM, Tony Stocker wrote:

> James,
> 
> Okay, we've tested this and everything works fine now.  I do have two
> quick questions:
> 
>       1.  Can you describe what we're doing "wrong" with our metadata?
>       I mean we're obviously doing something that you're not expecting
>       if you're having to maintain a code block with a comment with our
>       name in it.  I can't promise anything, but I'd at least like to
>       pass this along to the scientists and algorithm developers to see
>       if perhaps they could agree on making changes to make the metadata
>       more "friendly".
> 
>       2.  I tried to install the latest HDF5 handler while doing this
>       1.7.1 rpm install, but unfortunately the latest version of the
>       HDF 5 rpms is now 1.8.7 and your handler wants 1.8.5 only.  So my
>       question is, is there any way to make your handler rpms accept an
>       => (equal to or greater) with regards to the underlying library
>       dependency version?  Or do things absolutely have to be fixed to a
>       particular rpm version?
> 
> Thanks again for all the help!
> 
> Tony
> 
> 
> On Tue, 6 Sep 2011, Gallagher James wrote:
> 
>> 
>> On Sep 6, 2011, at 6:13 AM, Tony Stocker wrote:
>> 
>>> James,
>>> 
>>> Is there an updated package that we can install and use to see if the 
>>> original problem is fixed?
>> 
>> Yes - The Hyrax 1.7.1 has a new package for the HDF4 handler that should 
>> work as a drop in replacement. I did not test that for both the 32 and 
>> 64-bit linux builds, but it's likely to work. In the worst case, the changes 
>> outside of the hdf4 handler and the BES (but not a change to the BES's ABI).
>> 
>> Here's a link to the release: http://www.opendap.org/download/hyrax/1.7.1
>> 
>> Here's a link to the 64-bit Linux RPM for the hdf4 handler: 
>> http://www.opendap.org/pub/binary/hdf4_handler/centos5.2/3.9.3_x86_64
>> 
>> James
>> 
>>> 
>>> Tony
>>> 
>>> On Wed, 17 Aug 2011, Gallagher James wrote:
>>> 
>>>> 
>>>> On Aug 17, 2011, at 2:41 PM, Gallagher James wrote:
>>>> 
>>>>> 
>>>>> On Aug 17, 2011, at 12:50 PM, Tony Stocker wrote:
>>>>> 
>>>>>> 
>>>>>> James,
>>>>>> 
>>>>>> I've sent your write-up along to our algorithm developers.  I will say 
>>>>>> that this issue is present in all of our HDF files in version 7, 
>>>>>> regardless of algorithm.  Beyond that I'm afraid that as systems guy, 
>>>>>> I'm not in much of position to comment on the approach.
>>>>> 
>>>>> I'm looking at the hdf4 handler and there are number of issues there. One 
>>>>> is that the handler seems to think that every dimension of every array 
>>>>> must have a name, so those without a name in the data file get names 
>>>>> 'fakeDim_n' where n is 0, 1, 2, ... I'd like to silence that and also 
>>>>> remove the attributes that are also being created for those 'fake dims' 
>>>>> because it looks like those are only the 'fake dim' names and nothing 
>>>>> more. It's easy to use the API to read the names (or discover there's no 
>>>>> name associated with a particular dimension) from libdap or any one of 
>>>>> the other APIs.
>>>> 
>>>> Gads, I just found this comment in the handler code: // Reduce the fakeDim 
>>>> list. FakeDim is found to be used by TRMM team.
>>>> 
>>>> Comments?
>>>> 
>>>> PS. I have fixed the bug that prompted this email w/o resorting to 
>>>> removing the fakeDims or their 'name' attributes. I'd still like to remove 
>>>> the latter if possible (that is, remove the fakeDim attribute 'String name 
>>>> "fakeDim0"') but will wait on what you say.
>>>> 
>>>>> 
>>>>> Do you (and by extension NASA, THG) see any problem with my making the 
>>>>> handler behave a little differently so long as we can get back the old 
>>>>> behavior (I'll use a run-time parameter to switch on the new behavior)? 
>>>>> It may be that the new behavior is required to read some files. I'm not 
>>>>> sure about that last part, I have a fair amount more work to do on this 
>>>>> problem.
>>>>> 
>>>>> James
>>>>>> 
>>>>>> Tony
>>>>>> 
>>>>>>> On Tue, Aug 16, 2011 at 6:31 PM, Gallagher James <address@hidden> wrote:
>>>>>>>> 
>>>>>>>> On Aug 11, 2011, at 1:30 PM, Tony Stocker wrote:
>>>>>>>> 
>>>>>>>>> James,
>>>>>>>>> 
>>>>>>>>> Any word on this issue?
>>>>>>>> 
>>>>>>>> Tony,
>>>>>>>> 
>>>>>>>> Are the VData variables (I'm not sure if 'variables' is the right 
>>>>>>>> word) empty? When I look at the two kinds of files, in the v7 file 
>>>>>>>> there are a number of VData that, when viewed with vshow, look like:
>>>>>>>> 
>>>>>>>> vs:3 <1962/10340> nv=0 i=0 fld [SDS variable] vsize=4 (NoName {SDSVar})
>>>>>>>> 
>>>>>>>> That is the field is called 'SDS variable', the name is NoName and nv 
>>>>>>>> is 0. In the v6 files (the ones that work) there are no entries with 
>>>>>>>> nv=0; all of the nv values are >= 1 in the files that work.
>>>>>>>> 
>>>>>>>> In the documentation for VSsetfield() (which must be called before a 
>>>>>>>> Vread()) it says that if the VData is empty (I'm paraphrasing here) 
>>>>>>>> then the function will return -1 (the code for failure). However, 
>>>>>>>> prior to calling that function we are testing a number of other 
>>>>>>>> things. These VData have all of the following: a fieldname, 
>>>>>>>> fieldorder, fieldsize, and fieldtype.
>>>>>>>> 
>>>>>>>> I will try building the handler so that it treats this particular 
>>>>>>>> error as meaning the vdata is empty, but it's very hard for me to tell 
>>>>>>>> what's what in these very large HDF4 files. While I work on that, can 
>>>>>>>> you let me know if this seems to be a promising tack?
>>>>>>>> 
>>>>>>>> HDF folks: Can you weigh in on this? Am I close to understanding the 
>>>>>>>> semantics of VSsetfield()?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> James
>>>>>>>> 
>>>>>>>> PS. The ticket is http://scm.opendap.org:8090/trac/ticket/1793
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Tony
>>>>>>>>> 
>>>>>>>>> On Thu, 21 Jul 2011, address@hidden wrote:
>>>>>>>>> 
>>>>>>>>>> Tony,
>>>>>>>>>> I'm out for the next few days, but do send those files. The problem 
>>>>>>>>>> is in the hdf4 handler. Most likely anyway.
>>>>>>>>>> James
>>>>>>>>>> Sent from my Verizon Wireless Phone
>>>>>>>>>> ----- Reply message -----
>>>>>>>>>> From: "Tony Stocker" <address@hidden>
>>>>>>>>>> Date: Thu, Jul 21, 2011 2:40 pm
>>>>>>>>>> Subject: [support] Re: Could not read vdata records
>>>>>>>>>> To: "OPENDAP support" <address@hidden>
>>>>>>>>>> Cc: "Owen Kelley" <address@hidden>
>>>>>>>>>> And now to actually include the attachments (*sigh*)
>>>>>>>>>> On Thu, 21 Jul 2011, Tony Stocker wrote:
>>>>>>>>>>> Hi Guys,
>>>>>>>>>>> 
>>>>>>>>>>> Okay, quick and dirty description of the problem.  We have started
>>>>>>>>>>> reprocessing our science data, i.e. developers have new algorithms 
>>>>>>>>>>> and so we
>>>>>>>>>>> re-run everything.  Our old data is V6 and the new data is V7.  We 
>>>>>>>>>>> keep both
>>>>>>>>>>> around for comparison purposes.
>>>>>>>>>>> 
>>>>>>>>>>> OpenDAP has been working fine with our V6 data, and continues to do 
>>>>>>>>>>> so.
>>>>>>>>>>> 
>>>>>>>>>>> However it is choking on all of our V7 data, producing the error 
>>>>>>>>>>> message:
>>>>>>>>>>> "libdap exception building response: error_code = 1001: Could not 
>>>>>>>>>>> read Vdata
>>>>>>>>>>> records."
>>>>>>>>>>> 
>>>>>>>>>>> I'm going attaching two text files containing debug output of a 
>>>>>>>>>>> successful V6
>>>>>>>>>>> open and an unsuccessful V7 open of the same day/orbit:
>>>>>>>>>>> od.v6.19980101.1c21.00539-debug.txt i.e. WORKING FILE
>>>>>>>>>>> od.v7.19980101.1c21.00539-debug.txt i.e. NOT working FILE
>>>>>>>>>>> 
>>>>>>>>>>> I also wanted to see if you guys would let me send you both of 
>>>>>>>>>>> these HDF
>>>>>>>>>>> files (the v6 and v7 ones) to see if you guys see the same behavior 
>>>>>>>>>>> on your
>>>>>>>>>>> installations.  I haven't attached these to this email however 
>>>>>>>>>>> since each is
>>>>>>>>>>> ~150 MB in size.  I think it would be helpful to see if the problem 
>>>>>>>>>>> is
>>>>>>>>>>> universal or localized to our installation.  Let me know if you're 
>>>>>>>>>>> willing to
>>>>>>>>>>> take and test the files, and how you'd like to get them (email, 
>>>>>>>>>>> ftp, etc.)
>>>>>>>>>>> 
>>>>>>>>>>> I also want to mention that we have verified that the V7 HDF files 
>>>>>>>>>>> are not
>>>>>>>>>>> corrupt or otherwise unreadable in themselves by using two other HDF
>>>>>>>>>>> viewer/reader programs on them, and they come up fine, our only 
>>>>>>>>>>> problem seems
>>>>>>>>>>> to be with the OpenDAP server reading them.
>>>>>>>>>>> 
>>>>>>>>>>> Here are our software product versions:
>>>>>>>>>>> bes-3.9.0-1
>>>>>>>>>>> bes-devel-3.9.0-1
>>>>>>>>>>> dap-server-4.1.0-1
>>>>>>>>>>> libdap-3.11.0-1
>>>>>>>>>>> libdap-devel-3.11.0-1
>>>>>>>>>>> olfs-1.7.1
>>>>>>>>>>> freeform_handler-3.8.2-1
>>>>>>>>>>> hdf4_handler-3.9.1-1
>>>>>>>>>>> hdf5_handler-1.4.3-1
>>>>>>>>>>> netcdf_handler-3.9.2-1
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Look forward to hearing back from you guys, since this is 
>>>>>>>>>>> perplexing and
>>>>>>>>>>> pretty much out of our balliwick.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks!
>>>>>>>>>>> 
>>>>>>>>>>> Tony
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> +-----------------------------------+
>>>>>>>>>> |  Tony Stocker                     |
>>>>>>>>>> |  Systems Programmer               |
>>>>>>>>>> |  TSDIS/TRMM Code 610.2            |
>>>>>>>>>> |  301-614-5738 (office)            |
>>>>>>>>>> |  301-614-5269 (fax)               |
>>>>>>>>>> |  address@hidden            |
>>>>>>>>>> +-----------------------------------+
>>>>>>>>>> --
>>>>>>>>>> This message has been scanned for viruses and
>>>>>>>>>> dangerous content by MailScanner, and is
>>>>>>>>>> believed to be clean.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> +-----------------------------------+
>>>>>>>>> |  Tony Stocker                     |
>>>>>>>>> |  Systems Programmer               |
>>>>>>>>> |  TSDIS/TRMM Code 610.2            |
>>>>>>>>> |  301-614-5738 (office)            |
>>>>>>>>> |  301-614-5269 (fax)               |
>>>>>>>>> |  address@hidden            |
>>>>>>>>> +-----------------------------------+
>>>>>>>>> --
>>>>>>>>> This message has been scanned for viruses and
>>>>>>>>> dangerous content by MailScanner, and is
>>>>>>>>> believed to be clean.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> James Gallagher
>>>>>>>> jgallagher at opendap.org
>>>>>>>> 406.723.8663
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> +-----------------------------------+
>>>>>> |  Tony Stocker                     |
>>>>>> |  Systems Programmer               |
>>>>>> |  TSDIS/TRMM Code 610.2            |
>>>>>> |  301-614-5738 (office)            |
>>>>>> |  301-614-5269 (fax)               |
>>>>>> |  address@hidden            |
>>>>>> +-----------------------------------+
>>>>>> --
>>>>>> This message has been scanned for viruses and
>>>>>> dangerous content by MailScanner, and is
>>>>>> believed to be clean.
>>>>>> 
>>>>> 
>>>>> --
>>>>> James Gallagher
>>>>> jgallagher at opendap.org
>>>>> 406.723.8663
>>>>> 
>>>> 
>>>> --
>>>> James Gallagher
>>>> jgallagher at opendap.org
>>>> 406.723.8663
>>>> 
>>>> 
>>> 
>>> --
>>> +-----------------------------------+
>>> |  Tony Stocker                     |
>>> |  Systems Programmer               |
>>> |  TSDIS/TRMM Code 610.2            |
>>> |  301-614-5738 (office)            |
>>> |  301-614-5269 (fax)               |
>>> |  address@hidden            |
>>> +-----------------------------------+
>>> 
>>> --
>>> This message has been scanned for viruses and
>>> dangerous content by MailScanner, and is
>>> believed to be clean.
>>> 
>> 
>> --
>> James Gallagher
>> jgallagher at opendap.org
>> 406.723.8663
>> 
>> 
> 
> -- 
> +-----------------------------------+
> |  Tony Stocker                     |
> |  Systems Programmer               |
> |  TSDIS/TRMM Code 610.2            |
> |  301-614-5738 (office)            |
> |  301-614-5269 (fax)               |
> |  address@hidden            |
> +-----------------------------------+
> 
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
> 

--
James Gallagher
jgallagher at opendap.org
406.723.8663