ucar.nc2.ncml
Class Aggregation

java.lang.Object
  ucar.nc2.ncml.Aggregation

All Implemented Interfaces:

ProxyReader

Direct Known Subclasses:

AggregationOuterDimension, AggregationTiled, AggregationUnion

public abstract class Aggregation
extends java.lang.Object
implements ProxyReader

Superclass for NcML Aggregation.

Author:

caron

Nested Class Summary

protected class	Aggregation.DatasetProxyReader
static class	Aggregation.Type
protected static class	Aggregation.TypicalDataset

Field Summary

protected boolean	cacheDirty
protected DatasetCollectionManager	datasetManager
protected java.util.List<ucar.nc2.ncml.Aggregation.Dataset>	datasets
protected java.lang.String	dateFormatMark
protected boolean	debug
protected boolean	debugDateParse
protected boolean	debugOpenFile
protected boolean	debugProxy
protected boolean	debugRead
protected boolean	debugSyncDetail
protected java.lang.String	dimName
protected static DiskCache2	diskCache2
protected java.util.EnumSet<NetcdfDataset.Enhance>	enhance
protected static java.util.concurrent.Executor	executor
protected java.util.List<ucar.nc2.ncml.Aggregation.Dataset>	explicitDatasets
protected DateFormatter	formatter
protected boolean	isDate
protected static org.slf4j.Logger	logger
protected NetcdfDataset	ncDataset
protected java.lang.Object	spiObject
protected boolean	timeUnitsChange
protected Aggregation.Type	type
protected static Aggregation.TypicalDataset	typicalDatasetMode

Constructor Summary

protected

Aggregation(NetcdfDataset ncd, java.lang.String dimName, Aggregation.Type type, java.lang.String recheckS) Create an Aggregation for the given NetcdfDataset.

Method Summary

void	addCrawlableDatasetScan(org.jdom.Element crawlableDatasetElement, java.lang.String dirName, java.lang.String suffix, java.lang.String regexpPatternString, java.lang.String dateFormatMark, java.util.EnumSet<NetcdfDataset.Enhance> mode, java.lang.String subdirs, java.lang.String olderThan) Add a crawlable dataset scan
void	addDataset(ucar.nc2.ncml.Aggregation.Dataset nested)
void	addExplicitDataset(java.lang.String cacheName, java.lang.String location, java.lang.String ncoordS, java.lang.String coordValueS, java.lang.String sectionSpec, FileFactory reader, CancelTask cancelTask) Add a nested dataset, specified by an explicit netcdf element.
protected abstract void	buildDataset(CancelTask cancelTask) Call this to build the dataset objects
void	close() Release all resources associated with the aggregation
protected void	closeDatasets()
protected Variable	findVariable(NetcdfFile ncfile, Variable mainV)
void	finish(CancelTask cancelTask)
protected java.util.List<ucar.nc2.ncml.Aggregation.Dataset>	getDatasets()
java.lang.String	getDimensionName() Get dimension name to join on
protected java.lang.String	getLocation()
Aggregation.Type	getType() Get type of aggregation
protected ucar.nc2.ncml.Aggregation.Dataset	getTypicalDataset() Open one of the nested datasets as a template for the aggregation dataset.
protected ucar.nc2.ncml.Aggregation.Dataset	makeDataset(java.lang.String cacheName, java.lang.String location, java.lang.String ncoordS, java.lang.String coordValueS, java.lang.String sectionSpec, java.util.EnumSet<NetcdfDataset.Enhance> enhance, FileFactory reader) Dataset factory, so subclasses can override
protected void	makeDatasets(CancelTask cancelTask) Make the Dataset objects.
protected void	persistRead() read info from the persistent XML file, if it exists; overridden in AggregationExisting
void	persistWrite() Allow information to be make persistent.
abstract Array	read(Variable mainv, CancelTask cancelTask) Read an aggregation variable: A variable whose data spans multiple files.
abstract Array	read(Variable mainv, Section section, CancelTask cancelTask) Read a section of an aggregation variable.
protected abstract void	rebuildDataset() Call this when rescan has found changed datasets
protected void	setDatasetAcquireProxy(ucar.nc2.ncml.Aggregation.Dataset typicalDataset, NetcdfDataset newds) All non-agg variables use a proxy to acquire the file before reading.
static void	setExecutor(java.util.concurrent.Executor exec)
static void	setPersistenceCache(DiskCache2 dc)
static void	setTypicalDatasetMode(java.lang.String mode)
boolean	sync()
boolean	syncExtend(boolean force) Check to see if its time to rescan directory, and if so, rescan and extend dataset if needed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

typicalDatasetMode

protected static Aggregation.TypicalDataset typicalDatasetMode

logger

protected static org.slf4j.Logger logger

diskCache2

protected static DiskCache2 diskCache2

executor

protected static java.util.concurrent.Executor executor

ncDataset

protected NetcdfDataset ncDataset

type

protected Aggregation.Type type

spiObject

protected java.lang.Object spiObject

explicitDatasets

protected java.util.List<ucar.nc2.ncml.Aggregation.Dataset> explicitDatasets

datasets

protected java.util.List<ucar.nc2.ncml.Aggregation.Dataset> datasets

datasetManager

protected DatasetCollectionManager datasetManager

cacheDirty

protected boolean cacheDirty

dimName

protected java.lang.String dimName

timeUnitsChange

protected boolean timeUnitsChange

dateFormatMark

protected java.lang.String dateFormatMark

enhance

protected java.util.EnumSet<NetcdfDataset.Enhance> enhance

isDate

protected boolean isDate

formatter

protected DateFormatter formatter

debug

protected boolean debug

debugOpenFile

protected boolean debugOpenFile

debugSyncDetail

protected boolean debugSyncDetail

debugProxy

protected boolean debugProxy

debugRead

protected boolean debugRead

debugDateParse

protected boolean debugDateParse

Constructor Detail

Aggregation

protected Aggregation(NetcdfDataset ncd,
                      java.lang.String dimName,
                      Aggregation.Type type,
                      java.lang.String recheckS)

Create an Aggregation for the given NetcdfDataset. The following addXXXX methods are called, then finish(), before the object is ready for use.

Parameters:

ncd - Aggregation belongs to this NetcdfDataset

dimName - the aggregation dimension name

type - the Aggregation.Type

recheckS - how often to check if files have changes

Method Detail

setPersistenceCache

public static void setPersistenceCache(DiskCache2 dc)

setExecutor

public static void setExecutor(java.util.concurrent.Executor exec)

setTypicalDatasetMode

public static void setTypicalDatasetMode(java.lang.String mode)

addExplicitDataset

public void addExplicitDataset(java.lang.String cacheName,
                               java.lang.String location,
                               java.lang.String ncoordS,
                               java.lang.String coordValueS,
                               java.lang.String sectionSpec,
                               FileFactory reader,
                               CancelTask cancelTask)

Add a nested dataset, specified by an explicit netcdf element. enhance is handled by the reader, so its always false here.

Parameters:

cacheName - a unique name to use for caching

location - attribute "location" on the netcdf element

ncoordS - attribute "ncoords" on the netcdf element

coordValueS - attribute "coordValue" on the netcdf element

sectionSpec - attribute "section" on the netcdf element

reader - factory for reading this netcdf dataset

cancelTask - user may cancel, may be null

addDataset

public void addDataset(ucar.nc2.ncml.Aggregation.Dataset nested)

addCrawlableDatasetScan

public void addCrawlableDatasetScan(org.jdom.Element crawlableDatasetElement,
                                    java.lang.String dirName,
                                    java.lang.String suffix,
                                    java.lang.String regexpPatternString,
                                    java.lang.String dateFormatMark,
                                    java.util.EnumSet<NetcdfDataset.Enhance> mode,
                                    java.lang.String subdirs,
                                    java.lang.String olderThan)
                             throws java.io.IOException

Add a crawlable dataset scan

Parameters:

crawlableDatasetElement - defines a CrawlableDataset, or null

dirName - scan this directory

suffix - filter on this suffix (may be null)

regexpPatternString - include if full name matches this regular expression (may be null)

dateFormatMark - create dates from the filename (may be null)

mode - how should files be enhanced

subdirs - equals "false" if should not descend into subdirectories

olderThan - files must be older than this time (now - lastModified >= olderThan); must be a time unit, may ne bull

Throws:

java.io.IOException - if I/O error

getType

public Aggregation.Type getType()

Get type of aggregation

Returns:

type of aggregation

getDimensionName

public java.lang.String getDimensionName()

Get dimension name to join on

Returns:

dimension name or null if type union/tiled

close

public void close()
           throws java.io.IOException

Release all resources associated with the aggregation

Throws:

java.io.IOException - on error

closeDatasets

protected void closeDatasets()
                      throws java.io.IOException

Throws:

java.io.IOException

persistWrite

public void persistWrite()
                  throws java.io.IOException

Allow information to be make persistent. Overridden in AggregationExisting

Throws:

java.io.IOException - on error

persistRead

protected void persistRead()

read info from the persistent XML file, if it exists; overridden in AggregationExisting

finish

public void finish(CancelTask cancelTask)
            throws java.io.IOException

Throws:

java.io.IOException

getDatasets

protected java.util.List<ucar.nc2.ncml.Aggregation.Dataset> getDatasets()

makeDatasets

protected void makeDatasets(CancelTask cancelTask)
                     throws java.io.IOException

Make the Dataset objects.

Parameters:

cancelTask - user can cancel

Throws:

java.io.IOException - on i/o error

buildDataset

protected abstract void buildDataset(CancelTask cancelTask)
                              throws java.io.IOException

Call this to build the dataset objects

Parameters:

cancelTask - maybe cancel

Throws:

java.io.IOException - on read error

rebuildDataset

protected abstract void rebuildDataset()
                                throws java.io.IOException

Call this when rescan has found changed datasets

Throws:

java.io.IOException - on read error

syncExtend

public boolean syncExtend(boolean force)
                   throws java.io.IOException

Check to see if its time to rescan directory, and if so, rescan and extend dataset if needed. Note that this just calls sync(), so structural metadata may be modified (!!)

Parameters:

force - if true, always rescan even if time not expired

Returns:

true if directory was rescanned and dataset may have been updated

Throws:

java.io.IOException - on io error

sync

public boolean sync()
             throws java.io.IOException

Throws:

java.io.IOException

getLocation

protected java.lang.String getLocation()

getTypicalDataset

protected ucar.nc2.ncml.Aggregation.Dataset getTypicalDataset()
                                                       throws java.io.IOException

Open one of the nested datasets as a template for the aggregation dataset.

Returns:

a typical Dataset

Throws:

java.io.FileNotFoundException - if there are no datasets

java.io.IOException

read

public abstract Array read(Variable mainv,
                           CancelTask cancelTask)
                    throws java.io.IOException

Read an aggregation variable: A variable whose data spans multiple files.

Specified by:

read in interface ProxyReader

Parameters:

mainv - the aggregation variable

cancelTask - allow the user to cancel

Returns:

the data array

Throws:

java.io.IOException

read

public abstract Array read(Variable mainv,
                           Section section,
                           CancelTask cancelTask)
                    throws java.io.IOException,
                           InvalidRangeException

Read a section of an aggregation variable.

Specified by:

read in interface ProxyReader

Parameters:

mainv - the aggregation variable

cancelTask - allow the user to cancel

section - read just this section of the data, refers to aggregated Variable's section.

Returns:

the data array section

Throws:

java.io.IOException

InvalidRangeException - if section has incorrect rank or shape.

makeDataset

protected ucar.nc2.ncml.Aggregation.Dataset makeDataset(java.lang.String cacheName,
                                                        java.lang.String location,
                                                        java.lang.String ncoordS,
                                                        java.lang.String coordValueS,
                                                        java.lang.String sectionSpec,
                                                        java.util.EnumSet<NetcdfDataset.Enhance> enhance,
                                                        FileFactory reader)

Dataset factory, so subclasses can override

Parameters:

cacheName - a unique name to use for caching

location - attribute "location" on the netcdf element

ncoordS - attribute "ncoords" on the netcdf element

coordValueS - attribute "coordValue" on the netcdf element

sectionSpec - attribute "sectionSpec" on the netcdf element

enhance - open dataset in enhance mode

reader - factory for reading this netcdf dataset

Returns:

a Aggregation.Dataset

setDatasetAcquireProxy

protected void setDatasetAcquireProxy(ucar.nc2.ncml.Aggregation.Dataset typicalDataset,
                                      NetcdfDataset newds)
                               throws java.io.IOException

All non-agg variables use a proxy to acquire the file before reading. If the variable is caching, read data into cache now. If not caching, VariableEnhanced.setProxyReader() is called.

Parameters:

typicalDataset - read from a "typical dataset"

newds - containing dataset

Throws:

java.io.IOException - on i/o error

findVariable

protected Variable findVariable(NetcdfFile ncfile,
                                Variable mainV)