[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #AXF-489883]: THREDDS bug?



Hi Vega:

Theres a lot of bugs in the aggregation code, weve been trying to phase it out. 
You will likely see exceptions being thrown in the log files. If you could find 
those and send them to use we may be able to diagnose it. Are these netcdf 
files? Could you send us a link to one?

John

> Dear John,
> 
> we act like a Dissemination Unit, gathering files from other partners
> spread across Europe who upload files at different time during the day
> (everyday).
> So we configured TDS to scan (aggregation scan directive) the filesystem
> every 5 minutes.
> 
> In these days we noticed that the load on the server was rasing until it
> became unstable and the TDS service stop working. Analysis on the system
> showed that there were some files open by  TDS process (lsof) several
> times (i.e. more than 30) which remain open...forever?
> 
> Analisyng those files, we noticed that in effect the were not perfect,
> e.g. they were missing the "latitude" variable.
> 
> (hope it clarify)
> 
> We were wondering if it's normal, if there's a parameter we can set up,
> or if we can suggest an improvement in TDS e.g. setting up a timeout
> after which TDS force the closure of the file...
> 
> Thanks and best regards
> Vega
> 
> Il 21/05/2014 00:51, Unidata THREDDS Support ha scritto:
> > Hi Vega:
> >
> > Definitely a bug if files are being left open. What does "during the 
> > harvest (scan) process" mean?
> >
> > John
> >
> >
> >
> >> Dear all,
> >>
> >> as you may remember, I'm working @ CNR and I'm involved (among others)
> >> in the MyOcean project (http://www.myocean.eu/).
> >>
> >> We use TDS massively (30Tb of data more or less) but I'd like to point
> >> out a problem we're facing: if there're problematic files (i.e. files
> >> with missing variables like "latitude"), during the harvest (scan)
> >> process TDS does not close them. In this way the number of open files
> >> held by the process keeps growing until it makes the service unstable
> >> and we need to restart the process. Rising the max open files just
> >> postpone the problem... (the solution adopted at the time is to delete
> >> problematic files and then restart TDS).
> >>
> >> Is there any config feature we can set up? Could it be a "little" bug of
> >> TDS? TDS version used is : 4.3.20-SNAPSHOT - 20131104.1056
> >>
> >> Thanks and best regards
> >> Vega Forneris
> >>
> >>
> > Ticket Details
> > ===================
> > Ticket ID: AXF-489883
> > Department: Support THREDDS
> > Priority: Normal
> > Status: Open
> >
> 
> 


Ticket Details
===================
Ticket ID: AXF-489883
Department: Support THREDDS
Priority: Normal
Status: Closed