Re: [thredds] Proxy Error with large aggregation FMRC on Tomcat7/TDS 4.2.9

• To: Bob Simons <Bob.Simons@xxxxxxxx>
• Subject: Re: [thredds] Proxy Error with large aggregation FMRC on Tomcat7/TDS 4.2.9
• From: Rich Signell <rsignell@xxxxxxxx>
• Date: Mon, 21 Nov 2011 11:59:40 -0500

Bob,

My FMRC featureCollection contains the line:

   <update startup="true" rescan="0 30 6 * * ? *" trigger="allow"/>

and I thought the startup="true" meant that the metadata for the
aggregation would be generated 1st on server startup, not when the
first user stumbles on the aggregation.

But I could be wrong, as I don't see any explicit mention of what
"startup=true" means on the FeatureCollections page:

http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.2/reference/collections/FeatureCollections.html

In any case, I never got this message before:
 "Proxy Error: The proxy server received an invalid response from an
>> upstream server.

Do you think this is just triggered by the aggregation of metadata
taking too long?

Thanks,
Rich
On Mon, Nov 21, 2011 at 11:33 AM, Bob Simons <Bob.Simons@xxxxxxxx> wrote:
> This may or may not explain the problem:
>
> My understanding is:
> THREDDS uses what programmers call "lazy" instantiation, meaning that it
> doesn't initialize datasets proactively, it waits until they are requested.
>  Similarly, if a dataset needs to be updated, THREDDS doesn't update it
> until there is a request for the dataset.
> Also, when a dataset is being initialized or updated, the old version of the
> dataset is not available.  For datasets with lots of files, (re)initializing
> the dataset may take some time.
>
> So your initial request may be the initial request, which causes the dataset
> to be initialized.
>
> On 11/21/2011 7:47 AM, Rich Signell wrote:
>>
>> Thredds Folk,
>> I upgraded my server to Tomcat7/TDS4.2.9 and now I'm getting some
>> problems with large aggregations (specifically, and FMRC with 1500
>> files).
>>
>> When I restart tomcat and try to access the "best time series" dataset
>> for the first time, I get:
>>
>> "Proxy Error: The proxy server received an invalid response from an
>> upstream server.
>> The proxy server could not handle the request GET
>> /thredds/catalog/coawst_2_2/fmrc/catalog.html.
>> Reason: Error reading from remote server"
>>
>> If I then try to access the best time series again, it works.
>>
>> I'm curious whether others are seeing this and what the problem is.
>>
>> In the featureCollectionScan.log, I see this:
>>
>> [2011-11-21T09:50:32.117-0500] INFO
>> thredds.inventory.CollectionUpdater: Schedule startup scan for
>> Config{name='coawst_2_2',
>> spec='/usgs/vault0/coawst/coawst_2/Output/coawst_us_#yyyyMMdd_HH#.nc$',
>> olderThan='10 min', recheckAfter='null'
>>  UpdateConfig{startup=true, rescan='0 30 6 * * ? *', triggerOk=true}
>>  ProtoConfig{choice=Penultimate, change='null', param='null',
>> outerNcml='[Element:<netcdf [Namespace:
>> http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2]/>]',
>> cacheAll=true}
>>  FmrcConfig: regularize=false datasetTypes=[Best]} at Mon Nov 21
>> 09:51:02 EST 2011
>>
>> Does that look okay?
>>
>> Thanks,
>> Rich
>
> --
> Sincerely,
>
> Bob Simons
> IT Specialist
> Environmental Research Division
> NOAA Southwest Fisheries Science Center
> 1352 Lighthouse Ave
> Pacific Grove, CA 93950-2079
> Phone: (831)658-3205
> Fax:   (831)648-8440
> Email: bob.simons@xxxxxxxx
>
> The contents of this message are mine personally and
> do not necessarily reflect any position of the
> Government or the National Oceanic and Atmospheric
> Administration.
> <>< <>< <>< <>< <>< <>< <>< <>< <><
>



-- 
Dr. Richard P. Signell   (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598

• Follow-Ups:
  • Re: [thredds] Proxy Error with large aggregation FMRC on Tomcat7/TDS 4.2.9
    • From: Ethan Davis

• References:
  • [thredds] Proxy Error with large aggregation FMRC on Tomcat7/TDS 4.2.9
    • From: Rich Signell