Thanks, Sean! So, will binLimit will go away in TDS 5.0? Or is streaming
I/O still under development? Also, are you suggesting that with -Xmx in
place, there is no need to set a binLimit?
Cheers,
John
On Mon, May 4, 2020 at 3:22 AM Sean Arms <sarms@xxxxxxxx> wrote:
> Greetings John!
>
> On Wed, Apr 29, 2020 at 3:42 PM John Maurer <jmaurer@xxxxxxxxxx> wrote:
>
>> Is there an easy way to determine the binLimit setting for a remote TDS?
>>
>>
> Currently there isn't a way to get those limits a priori. We could
> probably add something that exposes those limits without the need to make a
> request, but even then, interpreting those limits can get tricky when
> compression is involved (see below).
>
>
>> Also, when binLimit is exceed in OPeNDAP, does it return an error
>> message? I know an ASCII request spits out a 403 message like the following:
>>
>>> Error {
>>> code = 403;
>>> message = "Request too big=1275173.181384 Mbytes, max=1000.0";
>>> };
>>>
>>> I guess how that message gets conveyed depends on the client, too (e.g.,
>> Python, Matlab, etc.)?
>>
>>
> A binary request over the max limit will contain a similar message body.
> It's up to the client to check the message and expose it to the user. There
> was a recent github issue over on the netCDF-C repository which I believe
> led to having the C library at least return the message body, so now that
> kind of message should be visible:
>
> https://github.com/Unidata/netcdf-c/issues/1667
>
> That issue also outlines the trickiness of compression I mention above.
>
> What would be even better is to fix the underlying issue that causes the
> need for having a limit on the server side. The performance section of the
> TDS docs mentions
>
> "The OPeNDAP-Java layer of the server currently has to read the entire
> data request into memory before sending it to the client (we hope to get a
> streaming I/O solution working eventually). Generally clients only request
> subsets of large files, but if you need to support large data requests,
> make sure that the -Xmx parameter above is set accordingly." (
> https://www.unidata.ucar.edu/software/tds/current/reference/Performance.html
> )
>
> It's on my long list of fires, but I need to get TDS 5 out the door before
> trying to address that one. On a related note, Unidata is hiring if you
> know of anyone looking!
>
>
> https://ucar.wd5.myworkdayjobs.com/en-US/UCAR_Careers/job/Foothills-Lab-4/THREDDS-Developer---Software-Engineer-II_REQ-2020-107-1
>
> Certainly a candidate with the right skillset could dive into
> this...although we might have to reopen the position shortly after (the
> code gets pretty complicated).
>
> On a related note, we're grappling with a TDS that returns all 0's when
>> binLimit is exceeded, rather than returning an error status. Anybody seen
>> this? I've seen mention online of a netcdf-c timeout issue?
>>
>
> I'd be interested in knowing more about this. Exceeding the binLimit
> should always result in a 403.
>
> I hope all i s well in Hawaii!
>
> Sean
>
>
>> Thanks for any insights!,
>> John Maurer
>> Data System Engineer
>> Pacific Islands Ocean Observing System (PacIOOS)
>> University of Hawaii at Manoa
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>
