Greetings John!
Sadly I don't think I'll be able to address the binLimit before 5. If
you have enough memory to burn, you can crank up the bin limit as
large as you'd like. However, if you are using java or python, you
could switch to using the cdmremote service, which does streaming i/o
using google protocol buffers. From a users perspective (with a
compatible library), it's a drop in replacement (functionality- and
use-wise) of the dap 2 server shipped with the TDS.
Cheers,
Sean
On Wed, May 6, 2020 at 1:27 PM John Maurer <jmaurer@xxxxxxxxxx> wrote:
>
> Thanks, Sean! So, will binLimit will go away in TDS 5.0? Or is streaming I/O
> still under development? Also, are you suggesting that with -Xmx in place,
> there is no need to set a binLimit?
> Cheers,
> John
>
> On Mon, May 4, 2020 at 3:22 AM Sean Arms <sarms@xxxxxxxx> wrote:
>>
>> Greetings John!
>>
>> On Wed, Apr 29, 2020 at 3:42 PM John Maurer <jmaurer@xxxxxxxxxx> wrote:
>>>
>>> Is there an easy way to determine the binLimit setting for a remote TDS?
>>>
>>
>> Currently there isn't a way to get those limits a priori. We could probably
>> add something that exposes those limits without the need to make a request,
>> but even then, interpreting those limits can get tricky when compression is
>> involved (see below).
>>
>>>
>>> Also, when binLimit is exceed in OPeNDAP, does it return an error message?
>>> I know an ASCII request spits out a 403 message like the following:
>>>>
>>>> Error {
>>>> code = 403;
>>>> message = "Request too big=1275173.181384 Mbytes, max=1000.0";
>>>> };
>>>
>>> I guess how that message gets conveyed depends on the client, too (e.g.,
>>> Python, Matlab, etc.)?
>>>
>>
>> A binary request over the max limit will contain a similar message body.
>> It's up to the client to check the message and expose it to the user. There
>> was a recent github issue over on the netCDF-C repository which I believe
>> led to having the C library at least return the message body, so now that
>> kind of message should be visible:
>>
>> https://github.com/Unidata/netcdf-c/issues/1667
>>
>> That issue also outlines the trickiness of compression I mention above.
>>
>> What would be even better is to fix the underlying issue that causes the
>> need for having a limit on the server side. The performance section of the
>> TDS docs mentions
>>
>> "The OPeNDAP-Java layer of the server currently has to read the entire data
>> request into memory before sending it to the client (we hope to get a
>> streaming I/O solution working eventually). Generally clients only request
>> subsets of large files, but if you need to support large data requests, make
>> sure that the -Xmx parameter above is set accordingly."
>> (https://www.unidata.ucar.edu/software/tds/current/reference/Performance.html)
>>
>> It's on my long list of fires, but I need to get TDS 5 out the door before
>> trying to address that one. On a related note, Unidata is hiring if you know
>> of anyone looking!
>>
>> https://ucar.wd5.myworkdayjobs.com/en-US/UCAR_Careers/job/Foothills-Lab-4/THREDDS-Developer---Software-Engineer-II_REQ-2020-107-1
>>
>> Certainly a candidate with the right skillset could dive into
>> this...although we might have to reopen the position shortly after (the code
>> gets pretty complicated).
>>
>>> On a related note, we're grappling with a TDS that returns all 0's when
>>> binLimit is exceeded, rather than returning an error status. Anybody seen
>>> this? I've seen mention online of a netcdf-c timeout issue?
>>
>>
>> I'd be interested in knowing more about this. Exceeding the binLimit should
>> always result in a 403.
>>
>> I hope all i s well in Hawaii!
>>
>> Sean
>>
>>>
>>> Thanks for any insights!,
>>> John Maurer
>>> Data System Engineer
>>> Pacific Islands Ocean Observing System (PacIOOS)
>>> University of Hawaii at Manoa
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web. Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>>
>>>
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> https://www.unidata.ucar.edu/mailing_lists/
