Greetings John!
On Wed, Apr 29, 2020 at 3:42 PM John Maurer <jmaurer@xxxxxxxxxx> wrote:
> Is there an easy way to determine the binLimit setting for a remote TDS?
>
>
Currently there isn't a way to get those limits a priori. We could probably
add something that exposes those limits without the need to make a request,
but even then, interpreting those limits can get tricky when compression is
involved (see below).
> Also, when binLimit is exceed in OPeNDAP, does it return an error message?
> I know an ASCII request spits out a 403 message like the following:
>
>> Error {
>> code = 403;
>> message = "Request too big=1275173.181384 Mbytes, max=1000.0";
>> };
>>
>> I guess how that message gets conveyed depends on the client, too (e.g.,
> Python, Matlab, etc.)?
>
>
A binary request over the max limit will contain a similar message body.
It's up to the client to check the message and expose it to the user. There
was a recent github issue over on the netCDF-C repository which I believe
led to having the C library at least return the message body, so now that
kind of message should be visible:
https://github.com/Unidata/netcdf-c/issues/1667
That issue also outlines the trickiness of compression I mention above.
What would be even better is to fix the underlying issue that causes the
need for having a limit on the server side. The performance section of the
TDS docs mentions
"The OPeNDAP-Java layer of the server currently has to read the entire data
request into memory before sending it to the client (we hope to get a
streaming I/O solution working eventually). Generally clients only request
subsets of large files, but if you need to support large data requests,
make sure that the -Xmx parameter above is set accordingly." (
https://www.unidata.ucar.edu/software/tds/current/reference/Performance.html
)
It's on my long list of fires, but I need to get TDS 5 out the door before
trying to address that one. On a related note, Unidata is hiring if you
know of anyone looking!
https://ucar.wd5.myworkdayjobs.com/en-US/UCAR_Careers/job/Foothills-Lab-4/THREDDS-Developer---Software-Engineer-II_REQ-2020-107-1
Certainly a candidate with the right skillset could dive into
this...although we might have to reopen the position shortly after (the
code gets pretty complicated).
On a related note, we're grappling with a TDS that returns all 0's when
> binLimit is exceeded, rather than returning an error status. Anybody seen
> this? I've seen mention online of a netcdf-c timeout issue?
>
I'd be interested in knowing more about this. Exceeding the binLimit should
always result in a 403.
I hope all i s well in Hawaii!
Sean
> Thanks for any insights!,
> John Maurer
> Data System Engineer
> Pacific Islands Ocean Observing System (PacIOOS)
> University of Hawaii at Manoa
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
