Re: [thredds] Proxy Error with large aggregation FMRC on Tomcat7/TDS 4.2.9

Hi Rich,

What kind of proxy setup do you have in front of your TDS server? I'm
surprised the proxy doesn't pass along whatever error it gets. Does it
take a while before you get the "Proxy Error" message? Perhaps the proxy
has a timeout limit and the response exceeds that limit.

The startup="true" should cause a scan on server startup. John might
have more thoughts on what is going on here.

What messages do you get for this request in the threddsServlet.log
file? And is there a message for this request in your access logs?

Ethan

On 11/21/2011 9:59 AM, Rich Signell wrote:
> Bob,
> 
> My FMRC featureCollection contains the line:
> 
>    <update startup="true" rescan="0 30 6 * * ? *" trigger="allow"/>
> 
> and I thought the startup="true" meant that the metadata for the
> aggregation would be generated 1st on server startup, not when the
> first user stumbles on the aggregation.
> 
> But I could be wrong, as I don't see any explicit mention of what
> "startup=true" means on the FeatureCollections page:
> 
> http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.2/reference/collections/FeatureCollections.html
> 
> In any case, I never got this message before:
>  "Proxy Error: The proxy server received an invalid response from an
>>> upstream server.
> 
> Do you think this is just triggered by the aggregation of metadata
> taking too long?
> 
> Thanks,
> Rich
> On Mon, Nov 21, 2011 at 11:33 AM, Bob Simons <Bob.Simons@xxxxxxxx> wrote:
>> This may or may not explain the problem:
>>
>> My understanding is:
>> THREDDS uses what programmers call "lazy" instantiation, meaning that it
>> doesn't initialize datasets proactively, it waits until they are requested.
>>  Similarly, if a dataset needs to be updated, THREDDS doesn't update it
>> until there is a request for the dataset.
>> Also, when a dataset is being initialized or updated, the old version of the
>> dataset is not available.  For datasets with lots of files, (re)initializing
>> the dataset may take some time.
>>
>> So your initial request may be the initial request, which causes the dataset
>> to be initialized.
>>
>> On 11/21/2011 7:47 AM, Rich Signell wrote:
>>>
>>> Thredds Folk,
>>> I upgraded my server to Tomcat7/TDS4.2.9 and now I'm getting some
>>> problems with large aggregations (specifically, and FMRC with 1500
>>> files).
>>>
>>> When I restart tomcat and try to access the "best time series" dataset
>>> for the first time, I get:
>>>
>>> "Proxy Error: The proxy server received an invalid response from an
>>> upstream server.
>>> The proxy server could not handle the request GET
>>> /thredds/catalog/coawst_2_2/fmrc/catalog.html.
>>> Reason: Error reading from remote server"
>>>
>>> If I then try to access the best time series again, it works.
>>>
>>> I'm curious whether others are seeing this and what the problem is.
>>>
>>> In the featureCollectionScan.log, I see this:
>>>
>>> [2011-11-21T09:50:32.117-0500] INFO
>>> thredds.inventory.CollectionUpdater: Schedule startup scan for
>>> Config{name='coawst_2_2',
>>> spec='/usgs/vault0/coawst/coawst_2/Output/coawst_us_#yyyyMMdd_HH#.nc$',
>>> olderThan='10 min', recheckAfter='null'
>>>  UpdateConfig{startup=true, rescan='0 30 6 * * ? *', triggerOk=true}
>>>  ProtoConfig{choice=Penultimate, change='null', param='null',
>>> outerNcml='[Element:<netcdf [Namespace:
>>> http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2]/>]',
>>> cacheAll=true}
>>>  FmrcConfig: regularize=false datasetTypes=[Best]} at Mon Nov 21
>>> 09:51:02 EST 2011
>>>
>>> Does that look okay?
>>>
>>> Thanks,
>>> Rich
>>
>> --
>> Sincerely,
>>
>> Bob Simons
>> IT Specialist
>> Environmental Research Division
>> NOAA Southwest Fisheries Science Center
>> 1352 Lighthouse Ave
>> Pacific Grove, CA 93950-2079
>> Phone: (831)658-3205
>> Fax:   (831)648-8440
>> Email: bob.simons@xxxxxxxx
>>
>> The contents of this message are mine personally and
>> do not necessarily reflect any position of the
>> Government or the National Oceanic and Atmospheric
>> Administration.
>> <>< <>< <>< <>< <>< <>< <>< <>< <><
>>
> 
> 
> 

-- 
Ethan Davis                                       UCAR Unidata Program
edavis@xxxxxxxxxxxxxxxx                    http://www.unidata.ucar.edu