Don,
An obvious test case would be a copy of a file before the update happens
and a copy of the same file just after the update happens. With those two
files John's group could simulate the update easily. :-)
Roland
On Sun, May 6, 2012 at 10:04 AM, Don Murray <don.murray@xxxxxxxx> wrote:
> John-
>
>
> On 5/5/12 4:56 PM, John Caron wrote:
>
>> We have someone working on trying to reproduce this problem. If we can
>> reproduce it, we can fix it. Any help doing that would be appreciated.
>>
>
> Since I'm the one who updates the dataset, here's what happens. The SST
> datasets consist of yearly files which we are trying to aggregate into one
> dataset. The latest file (or files depending if we are crossing a year
> boundary) is updated on a daily basis and the last 14 days of data are
> overwritten and a new timestep (day) is added. We do more than just add a
> timestep because the last 14 days of data are preliminary and subject to
> updates. The file(s) being updated are then synced out to the filesystem
> that TDS looks at.
>
> Hoop has said he would provide any catalog/configuration information you
> require. If it would speed up the process, perhaps someone from Unidata
> could drive across town and visit NOAA.
>
> Don
>
>
> On 5/4/2012 5:41 PM, Roy Mendelssohn wrote:
>>
>>> Hi All:
>>>
>>> I have been corresponding with Hoop off list, because the datasets he
>>> is having problems serving are important to NOAA, and getting them
>>> served properly is important. His problem is:
>>>
>>> On 05/02/2012 04:28 PM, Hoop wrote:
>>>>>
>>>>>> All,
>>>>>>
>>>>>> This is my latest in a now monthly series of requests for help with
>>>>>> doing aggregations with our TDS. The problem I first reported back
>>>>>> on 23 February, wherein aggregations don't notice time steps added
>>>>>> to the final file in the time series, is unresolved.. Since I last
>>>>>> wrote (4 April), we upgraded to 4.2.10. There was no effect that we
>>>>>> could discern. Whether we use NcML or FeatureCollection, new time
>>>>>> steps in the final file go unnoticed until Tomcat is restarted.
>>>>>> Fabulously, if a new file is added without restarting Tomcat, the
>>>>>> initial time steps in the new final file are added to the aggregation,
>>>>>> leaving a gap where the time steps added to the previous "final" file
>>>>>> since the last Tomcat restart should be. This leads to complaints of
>>>>>> the aggregation not being CF-compliant, since it appears to have
>>>>>> uneven spacing in time.
>>>>>>
>>>>>> Interestingly, doing the aggregation in RAMADDA works as we would
>>>>>> expect, since it is frequently rebuilding the aggregation. So, while
>>>>>> it is perhaps less efficient than TDS, at least it is reliable.
>>>>>>
>>>>>> -Hoop
>>>>>>
>>>>>>
>>> I just checked with my group, and we have several aggregated datasets
>>> that work this way. We have tons of datasets where each file is a new
>>> time period, and they get updated just fine. But the ones where we are
>>> aggregating over files with multiple time periods, and one of those
>>> files is being updated with new data, the aggregation only gets
>>> updated by restarting tomcat. This has not been a major issue because
>>> our datasets are updated monthly, so we do it once and all is good.
>>> But if we were updating regularly, this would be a major issue, as it
>>> appears to be for Hoop's group. Mainly I am writing for three reasons:
>>>
>>> 1. This problem is not isolated to Hoop's group or how they are setup.
>>>
>>> 2. If anyone has this kind of aggregation updating successfully, can
>>> they point to the URL, post the catalog.xml and threddsConfig.xml, and
>>> perhaps sample files so we can test it and see what is different.
>>>
>>> 3. If successful examples can't be found that we can build on, is
>>> there anyway this can elevated in importance at Unidata? I know there
>>> is a lot being worked on, and priorities need to be set, but if a
>>> solution can't be found given the present code base, it strikes me as
>>> a serious bug.
>>>
>>> Thanks
>>>
>>> -Roy
>>>
>>>
>>>
>>> **********************
>>> "The contents of this message do not reflect any position of the U.S.
>>> Government or NOAA."
>>> **********************
>>> Roy Mendelssohn
>>> Supervisory Operations Research Analyst
>>> NOAA/NMFS
>>> Environmental Research Division
>>> Southwest Fisheries Science Center
>>> 1352 Lighthouse Avenue
>>> Pacific Grove, CA 93950-2097
>>>
>>> e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
>>> voice: (831)-648-9029
>>> fax: (831)-648-8440
>>> www: http://www.pfeg.noaa.gov/
>>>
>>> "Old age and treachery will overcome youth and skill."
>>> "From those who have been given much, much will be expected"
>>> "the arc of the moral universe is long, but it bends toward justice"
>>> -MLK Jr.
>>>
>>> ______________________________**_________________
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/**mailing_lists/<http://www.unidata.ucar.edu/mailing_lists/>
>>>
>>
>> ______________________________**_________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/**mailing_lists/<http://www.unidata.ucar.edu/mailing_lists/>
>>
>
> --
> Don Murray
> NOAA/ESRL/PSD and CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/**people/don.murray/<http://www.esrl.noaa.gov/psd/people/don.murray/>
>
>
> ______________________________**_________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/**mailing_lists/<http://www.unidata.ucar.edu/mailing_lists/>
>
