[thredds] over written file has old global attributes

Wed Nov 7 14:17:45 MST 2007

Hi Jean,

The TDS data file cache holds a pointer to each cached file along with 
all the metadata for the file. The attributes get pulled from the cache 
while data requests access the file on disk. The cache is supposed to 
check the last modified date for the file and re-read if it has been 
modified since the cache was last updated. But what you are reporting 
sure makes it sound like that capability (checking last modified) might 
be broken.

Just to make sure, next time you see this behavior, could you take a 
look at the cache on your server and, if the file is listed, clear the 
cache and see if the problem goes away? You have to have TDS Remote 
Management setup (see the TDS Reference for details, 
http://www.unidata.ucar.edu/projects/THREDDS/tech/reference/).

It will be a few weeks before we will be able to look at this in depth. 
For now, we have a new 3.16 release with a minor tweak that turns off 
caching if the max number of cached items is set to zero. You set the 
max values by editing your threddsConfig.xml file which is in the 
content/thredds directory. There are three types of caching you can 
adjust: NetcdfFileCache, NetcdfDatasetCache, and HTTPFileCache. You may 
only need to modify the NetcdfFileCache settings but NetcdfDatasetCache 
may come into play if you have datasets described with NcML. To change 
the settings, find the elements of the same name in your 
threddsConfig.xml and set max to 0.

You probably are not seeing this behavior every time because the cache 
occasionally scours out files that aren't as active as others.

Hope that helps,

Ethan

Jean.C.Newman at noaa.gov wrote:
> On the sift machine, because we were doing testing, we have "opendap" with 
> & without thredds and "looking" at the file - ncdump & dncdump with 
> "opendap no thredds" both give correct attributes while dcdump with 
> "thredds" gives old attribute information. Jean
>
> ncdump -h crescent_runup_ha.nc
> ...
> // global attributes:
>                  :history = "MOST v2.080 11-05-2007  16:09:22" ;
>                  :title = "Aleutian-Cascadia: Mwt 8.4, 21.00*b65" ;
>
> dncdump -h 
> http://sift.pmel.noaa.gov/thredds/dodsC/data/crescent_runup_ha.nc
> ...
> // global attributes:
>                  :history = "MOST v2.080 11-05-2007  11:49:33" ;
>                  :title = "Aleutian-Cascadia: Mwt 8.5, 29.00*b25" ;
>
> dncdump -h http://sift.pmel.noaa.gov/opendap/crescent_runup_ha.nc
> ...
> // global attributes:
>                  :history = "MOST v2.080 11-05-2007  16:09:22" ;
>                  :title = "Aleutian-Cascadia: Mwt 8.4, 21.00*b65" ;
>   ___________________________________________________________
>    Jean Newman                             Tel: 206-526-6531
>    NOAA Center for Tsunami Research
>    NOAA/PMEL/OERD2 - UW/JISAO              FAX: 206-526-6485
>    7600 Sand Point Way NE, Bldg. 3    jean.c.newman at noaa.gov
>    Seattle, WA 98115-6349               jcn at u.washington.edu
>   _________________________. URL: http://nctr.pmel.noaa.gov/
>                           (__________________________________
>
> On Tue, 6 Nov 2007, Jean.C.Newman at noaa.gov wrote:
>
>   
>> One other thing, the "old attributes" lasts. I made a run yesterday at
>> 11:23 am and it was over written several time the last at 16:09 and today
>> if I look at that output file via LAS/"dncdump -h" it still shows the "old
>> stuff" from the 11:23 run. "ncdump -h crescent_runup_ha.nc" has the
>> correct 16:09 information. Thanks, Jean
>>  ___________________________________________________________
>>   Jean Newman                             Tel: 206-526-6531
>>
>> On Tue, 6 Nov 2007, Jean.C.Newman at noaa.gov wrote:
>>
>>     
>>> The question:
>>> Is there some way to turn off caching of metadata so the file is checked
>>> every time it's accessed? Jean
>>>  ___________________________________________________________
>>>   Jean Newman                             Tel: 206-526-6531
>>>
>>> On Tue, 6 Nov 2007, Jean.C.Newman at noaa.gov wrote:
>>>
>>>       
>>>> It's not a browser caching problem. I have a very old LAS that accesses
>>>> these files. From the LAS, input for a model run is set up then a servlet
>>>> is run on 1 of several possible machines that starts a FORTRAN model run
>>>> that overwwites files so that the LAS interface has static files/"DODs"
>>>> URLs for the output. We are moving to different code but need to keep this
>>>> working until the new is up and reliable. The data updates, I can see
>>>> that in the plots, but I use global attributes in labeling and I get the
>>>> old information sometimes, 2 out of 31 SIM runs. The example I put out on
>>>> the sift machine was run on an internal machine and then sftp-ed to sift.
>>>> Then I made another run and again sftp-ed it to the sift machine and saw
>>>> the same "old attributes" problem, which I really didn't think I would.
>>>> If tomcat is restarted then I get the new information so I imagine it is
>>>> some caching problem. I did try removing the file before rerunning the
>>>> SIM and that did not solve the problem. Is there some way to turn off
>>>> caching of metadata so the file is checked every time it's accessed? Jean
>>>>  ___________________________________________________________
>>>>   Jean Newman                             Tel: 206-526-6531
>>>>         

-- 
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    edavis at ucar.edu
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------