Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Yes,because the JoinExisting aggregation needs to open each file to know the corresponding time coordinate value. As you mention the trade-off is that the ncml could be big and possibly you could get some issues when the ncml has too many elements.
The other solution could be use what is mentioned in the Aggregation tutotirial: If there is exactly one time slice in each file of the JoinExisting aggregation, and you are using a scan element to dynamically scan the files in a directory, then you can use the dateFormatMark attribute to derive the date from the filename.
but it depends on the layout of your dataset.
Regards
Antonio
--
Antonio S. Cofiño
Grupo de Meteorología de Santander
Dep. de Matemática Aplicada y
Ciencias de la Computación
Universidad de Cantabria
http://www.meteo.unican.es
El viernes, 20 de diciembre de 2013 21:47:41, Kevin Manross escribió:
Thanks Antonio! I'll definitely give this idea a shot. Is there any performance hit if listing several thousand files in the catalog (as opposed to scanning the directory)? Thanks again! -kevin. On 12/20/13 1:34 PM, "Antonio S. Cofiño" wrote:Kevin, To improve the JoinExisting aggregation you can substitute the inner scan element by adding explicitly files (explicitly) you want aggregate and add the ncoords or the coordValue attribute to the netcdf element as it's been explained in the "Defining coordinates on a JoinExisting aggregation" section of the Aggregation document: http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/ncml/v2.2/Aggregation.html Be sure the aggregation cache for the TDS config is configured: http://www.unidata.ucar.edu/software/thredds/current/tds/tds4.3/reference/ThreddsConfigXMLFile.html#AggregationCache I hope this help. Regards Antonio -- Antonio S. Cofiño Grupo de Meteorología de Santander Dep. de Matemática Aplicada y Ciencias de la Computación Universidad de Cantabria http://www.meteo.unican.es El 20/12/2013 19:28, Kevin Manross escribió:Seasons Greetings! I really wish we didn't have these restrictions on data, but that's what I'm dealing with so please bear with me. We have some large (33 Tb, 840 Gb, etc) netCDF datasets that I am trying to aggregate. Many are in "time series" layout (I.e., single parameter grid spread out across many time steps [files], such as u10/u10_RCPP_2004_11.nc, u10/u10_RCPP_2004_12.nc, etc.) I initially tried a large nested aggregations such as: <dataset name="ds601.0-Agg" ID="ds601.0-AGG" & nbsp;&nbs p; urlPath="ds601.0/10/best" harvest="true"> <metadata inherited="true"> <serviceName>all</serviceName> <dataFormat>NetCDF</dataFormat> <dataType>GRID</dataType> </metadata> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> <!--attribute name="title" type="string" value="20th Century Simulation Yearly Timeseries-Parameter Aggregations"/--> <aggregation type="Union"> <netcdf> <aggregation dimName="time" type="joinExisting"> <scan location="/glade/p/rda/data/ds601.0/RCPP/1995_2005/glw/" suffix=".nc" subdirs="true"/> </aggregation> </netcdf> <netcdf> <aggregation dimName="time" type="joinExisting"> &n bsp; ; <scan location="/glade/p/rda/data/ds601.0/RCPP/1995_2005/graupel/" suffix=".nc" subdirs="true"/> </aggregation> </netcdf> <netcdf> <aggregation dimName="time" type="joinExisting"> <scan location="/glade/p/rda/data/ds601.0/RCPP/1995_200 5/olr/" suffix=".nc" subdirs="true"/> </aggregation> </netcdf> <netcdf> &nb sp; <aggregation dimName="time" type="joinExisting"> <scan location="/glade/p/rda/data/ds601.0/RCPP/1995_2005/psfc/" suffix=".nc" subdirs="true"/> </aggregation> </netcdf> ... ... ... </aggregation> </netcdf> </dataset> This takes a long time to build the cache file, and upon each revisit it goes through the process of rebuilding the file. Honestly, it is unusable this way from a user standpoint. However, everything works with the restrictions I have set up via Tomcat DataSourceRealm and webapps/thredds/WEB-INF/web.xml Mike McDonald had a really slick way to aggregate and cache the parameter timeseries files, and then build the union on demand. (see his response to the thread '"Too Many Open Files" Error. Dataset too big?' on 28 October 2013) . So using his example, I reformatted my catalog as such: <dataset name="Full Aggregation of ds601.0" ID="ds601.0-AGG" urlPath="aggregations/ds601.0/10/best" harvest="true"> <serviceName>all</serviceName> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> <aggregation type="Union"> <netcdf location="dods://localhost:8080/thredds/dodsC/internal/ds601.0/101/glw"/> <netcdf location="dods://localhost:8080/thredds/dodsC/internal/ds601.0/102/graupel"/> <netcdf location="dods://localhost:8080/thredds/dodsC/internal/ds601.0/103/olr"/> <netcdf location="dods://localhost:8080/thredds/dodsC/internal/ds601.0/104/psfc"/> ... ... ... </aggregation> </netcdf> </dataset> <dataset name="internal/ds601.0 Aggregation (glw)" ID="internal/ds601.0/101/glw" urlPath="internal/ds601.0/101/glw" harvest="true"> <serviceName>all</serviceName> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> <aggregation dimName="time" type="joinExisting"> <scan location="/data/glade/p/rda/data/ds601.0/RCPP/1995_2005/glw/" suffix=".nc" subdirs="true"/> </aggregation> </netcdf> </dataset> <dataset name="internal/ds601.0 Aggregation (graupel)" ID="internal/ds601.0/102/graupel" urlPath="internal/ds601.0/102/graupel" harvest="true"> <serviceName>all</serviceName> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> <aggregation dimName="time" type="joinExisting"> <scan location="/data/glade/p/rda/data/ds601.0/RCPP/1995_2005/graupel/" suffix=".nc" subdirs="true"/> </aggregation> </netcdf> </dataset> <dataset name="internal/ds601.0 Aggregation (olr)" ID="internal/ds601.0/103/olr" urlPath="internal/ds601.0/103/olr" harvest="true"> <serviceName>all</serviceName> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> <aggregation dimName="time" type="joinExisting"> <scan location="/data/glade/p/rda/data/ds601.0/RCPP/1995_2005/olr/" suffix=".nc" subdirs="true"/> </aggregation> </netcdf> </dataset> <dataset name="internal/ds601.0 Aggregation (psfc)" ID="internal/ds601.0/104/psfc" urlPath="internal/ds601.0/104/psfc" harvest="true"> <serviceName>all</serviceName> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> <aggregation dimName="time" type="joinExisting"> <scan location="/data/glade/p/rda/data/ds601.0/RCPP/1995_2005/psfc/" suffix=".nc" subdirs="true"/> </aggregation> </netcdf> </dataset> ... ... ... This sped things up immensely and the server is very responsive, however, I can't seem to get the authorization to work with the internal Union aggregation. I have attempted a number of things, such as: + https://www.unidata.ucar.edu/software/thredds/current/tds/reference/RestrictedAccess.html - 2. Restrict by Dataset using TDS Catalog for each joinExisting aggregation + Adding a valid username/password to the url in the netcdf location value of the Union call <aggregation type="Union"> <netcdf location="dods://USERNAME:PASSWORD@localhost:8080/thredds/dodsC/internal/ds601.0/101/glw"/> + trying the above with an http:// protocol The only thing that seems to work is to leave the the joinExisting aggregations unrestricted, but keep the restriction on the Union aggregation. I would like to do any of the following: 1) Hide the joinExisting aggregations (links) from the web browser 2) Since the joinExisting aggregations are only needed to populate the Union aggregation "internally" to the TDS, somehow ease restrictions when called within the TDS on the localhost 3) Somehow authorize the joinExisting aggregations within the Uinion aggregation 4) Hear of an alternative way to efficiently aggregate the timeseries parameters and then combine those aggregated timeseries. If this is completely undo-able, that is also helpful information, and I'll leave the aggregated timeseries (joinExisting) unrestricted. -kevin. -- Kevin Manross NCAR/CISL/Data Support Section Phone: (303)-497-1218 Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx> Web:http://rda.ucar.edu _______________________________________________ thredds mailing list thredds@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/-- Kevin Manross NCAR/CISL/Data Support Section Phone: (303)-497-1218 Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx> Web:http://rda.ucar.edu
thredds archives: