ucar.nc2.ncml
Class Aggregation

java.lang.Object
  ucar.nc2.ncml.Aggregation

All Implemented Interfaces:

ProxyReader

Direct Known Subclasses:

AggregationExisting, AggregationFmr, AggregationFmrc, AggregationNew, AggregationUnion

public abstract class Aggregation
extends java.lang.Object
implements ProxyReader

Implement NcML Aggregation.

Implementation Notes

Caching

• Case 1. Explicit list / Scan static directories (recheck=null)
  • A. AggCaching - keep track of ncoords, coordValues for joinExisting. Read on open, write on close. Could skip scan if cache exists.
  • B. NetcdfFileCache - write on close if changed (only first time). On sync, recheck = null means wont be reread.
• Case 2. Scan dynamic directories (recheck non-null)
  • A. AggCaching - keep track of ncoords, coordValues for joinExisting. Read on open, write on close. Could skip scan if cache exists, and recheck time not expired.
  • B. NetcdfFileCache - write on close if changed. On sync, if recheck time, then rescan.

Aggregation Coordinate Variable (aggCoord) Processing

Construction:

1. The aggregation element is processed first.
2. agg.finish() is called.
3. If the user has defined the aggCoord in the NcML, it is then processed, overriding whatever the aggregation has constructed. If values are defined, they are cached in the new variable.

Data Reading:

1. If values are cached, agg.read() is never called.
2. Each Dataset may have a coordinate value(s) defined in the NcML coordValue attribute.
3. If not, the coordinate value(s) is cached when the dataset is opened.
4. agg.read() uses those if they exist, else reads and caches.

Version:

$Revision: 69 $ $Date: 2006-07-13 00:12:58Z $

Author:

caron

Nested Class Summary

class	Aggregation.Dataset Encapsolates a NetcdfFile that is a component of the aggregation.
protected class	Aggregation.DatasetProxyReader
protected class	Aggregation.DirectoryScan Encapsolate a "scan" or "scan2" element: a directory that we want to scan.
protected class	Aggregation.MyFile Encapsolate a file that was scanned.
protected class	Aggregation.MyReplaceVariableCheck
static class	Aggregation.Type

Field Summary

protected boolean	debug
protected boolean	debugCacheDetail
protected boolean	debugOpenFile
protected boolean	debugProxy
protected boolean	debugRead
protected boolean	debugScan
protected boolean	debugSyncDetail
protected java.lang.String	dimName
protected static DiskCache2	diskCache2
protected java.util.List<Aggregation.Dataset>	explicitDatasets
protected DateFormatter	formatter
protected boolean	isDate
protected long	lastChecked
protected static org.slf4j.Logger	logger
protected NetcdfDataset	ncDataset
protected java.util.List<Aggregation.Dataset>	nestedDatasets
protected TimeUnit	recheck
protected java.util.List<Aggregation.DirectoryScan>	scanList
protected java.lang.Object	spiObject
protected Aggregation.Type	type
protected static int	TYPICAL_DATASET_LATEST
protected static int	TYPICAL_DATASET_PENULTIMATE
protected static int	TYPICAL_DATASET_RANDOM
protected static int	typicalDatasetMode
protected boolean	wasChanged

Constructor Summary

protected

Aggregation(NetcdfDataset ncd, java.lang.String dimName, Aggregation.Type type, java.lang.String recheckS) Create an Aggregation for the given NetcdfDataset.

Method Summary

void	addDataset(Aggregation.Dataset nested)
void	addDirectoryScan(java.lang.String dirName, java.lang.String suffix, java.lang.String regexpPatternString, java.lang.String dateFormatMark, java.lang.String enhance, java.lang.String subdirs, java.lang.String olderThan) Add a scan elemnt
void	addExplicitDataset(java.lang.String cacheName, java.lang.String location, java.lang.String ncoordS, java.lang.String coordValueS, NetcdfFileFactory reader, CancelTask cancelTask) Add a nested dataset (other than a union), specified by an explicit netcdf element.
void	addVariable(java.lang.String varName) Add a name from a variableAgg element
protected void	buildCoords(CancelTask cancelTask)
protected abstract void	buildDataset(boolean isNew, CancelTask cancelTask)
void	close() Release all resources associated with the aggregation
void	finish(CancelTask cancelTask)
DataType	getCoordinateType() What is the data type of the aggregation coordinate ?
java.lang.String	getDimensionName()
java.util.List<Aggregation.Dataset>	getNestedDatasets()
int	getTotalCoords()
Aggregation.Type	getType()
protected Aggregation.Dataset	getTypicalDataset() Open one of the nested datasets as a template for the aggregation dataset.
java.util.List	getVariables() Get the list of aggregation variables: variables whose data spans multiple files.
boolean	isDate()
protected Aggregation.Dataset	makeDataset(java.lang.String cacheName, java.lang.String location, java.lang.String ncoordS, java.lang.String coordValueS, boolean enhance, NetcdfFileFactory reader) Dataset factory, so subclasses can override
protected void	makeProxies(Aggregation.Dataset typicalDataset, NetcdfDataset newds)
void	persist() Overriden in AggregationExisting
protected void	persistRead()
Array	read(Variable mainv, CancelTask cancelTask) Read an aggregation variable: A variable whose data spans multiple files.
Array	read(Variable mainv, CancelTask cancelTask, java.util.List section) Read a section of an aggregation variable.
protected Array	readAggCoord(Variable aggCoord, CancelTask cancelTask)
protected Array	readAggCoord(Variable aggCoord, CancelTask cancelTask, java.util.List section)
protected boolean	rescan()
protected void	scan(java.util.List<Aggregation.Dataset> result, CancelTask cancelTask) Scan the directory(ies) and create nested Aggregation.Dataset objects.
static void	setPersistenceCache(DiskCache2 dc)
static void	setTypicalDatasetMode(java.lang.String mode)
boolean	sync()
protected void	syncDataset(CancelTask cancelTask)
boolean	syncExtend(boolean force) Check to see if its time to rescan directory, and if so, rescan and extend dataset if needed.
protected boolean	timeToRescan() Rescan if recheckEvery time has passed

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TYPICAL_DATASET_RANDOM

protected static int TYPICAL_DATASET_RANDOM

TYPICAL_DATASET_LATEST

protected static int TYPICAL_DATASET_LATEST

TYPICAL_DATASET_PENULTIMATE

protected static int TYPICAL_DATASET_PENULTIMATE

typicalDatasetMode

protected static int typicalDatasetMode

logger

protected static org.slf4j.Logger logger

diskCache2

protected static DiskCache2 diskCache2

ncDataset

protected NetcdfDataset ncDataset

dimName

protected java.lang.String dimName

type

protected Aggregation.Type type

nestedDatasets

protected java.util.List<Aggregation.Dataset> nestedDatasets

spiObject

protected java.lang.Object spiObject

explicitDatasets

protected java.util.List<Aggregation.Dataset> explicitDatasets

scanList

protected java.util.List<Aggregation.DirectoryScan> scanList

recheck

protected TimeUnit recheck

lastChecked

protected long lastChecked

wasChanged

protected boolean wasChanged

isDate

protected boolean isDate

formatter

protected DateFormatter formatter

debug

protected boolean debug

debugOpenFile

protected boolean debugOpenFile

debugCacheDetail

protected boolean debugCacheDetail

debugSyncDetail

protected boolean debugSyncDetail

debugProxy

protected boolean debugProxy

debugScan

protected boolean debugScan

debugRead

protected boolean debugRead

Constructor Detail

Aggregation

protected Aggregation(NetcdfDataset ncd,
                      java.lang.String dimName,
                      Aggregation.Type type,
                      java.lang.String recheckS)

Create an Aggregation for the given NetcdfDataset. The following addXXXX methods are called, then finish(), before the object is ready for use.

Parameters:

ncd - Aggregation belongs to this NetcdfDataset

dimName - the aggregation dimension name

type - the Aggregation.Type

recheckS - how often to check if files have changes

Method Detail

setPersistenceCache

public static void setPersistenceCache(DiskCache2 dc)

setTypicalDatasetMode

public static void setTypicalDatasetMode(java.lang.String mode)

addExplicitDataset

public void addExplicitDataset(java.lang.String cacheName,
                               java.lang.String location,
                               java.lang.String ncoordS,
                               java.lang.String coordValueS,
                               NetcdfFileFactory reader,
                               CancelTask cancelTask)

Add a nested dataset (other than a union), specified by an explicit netcdf element. enhance is handled by the reader, so its always false here.

Parameters:

cacheName - a unique name to use for caching

location - attribute "location" on the netcdf element

ncoordS - attribute "ncoords" on the netcdf element

coordValueS - attribute "coordValue" on the netcdf element

reader - factory for reading this netcdf dataset

cancelTask - user may cancel, may be null

addDataset

public void addDataset(Aggregation.Dataset nested)

addDirectoryScan

public void addDirectoryScan(java.lang.String dirName,
                             java.lang.String suffix,
                             java.lang.String regexpPatternString,
                             java.lang.String dateFormatMark,
                             java.lang.String enhance,
                             java.lang.String subdirs,
                             java.lang.String olderThan)
                      throws java.io.IOException

Add a scan elemnt

Parameters:

dirName - scan this directory

suffix - filter on this suffix (may be null)

regexpPatternString - include if full name matches this regular expression (may be null)

dateFormatMark - create dates from the filename (may be null)

enhance - should files bne enhanced?

subdirs - equals "false" if should not descend into subdirectories

olderThan - files must be older than this time (now - lastModified >= olderThan); must be a time unit, may ne bull

Throws:

java.io.IOException - if I/O error

addVariable

public void addVariable(java.lang.String varName)

Add a name from a variableAgg element

Parameters:

varName - name of variable to add

getDimensionName

public java.lang.String getDimensionName()

getTotalCoords

public int getTotalCoords()

getNestedDatasets

public java.util.List<Aggregation.Dataset> getNestedDatasets()

getType

public Aggregation.Type getType()

isDate

public boolean isDate()

getVariables

public java.util.List getVariables()

Get the list of aggregation variables: variables whose data spans multiple files.

getCoordinateType

public DataType getCoordinateType()

What is the data type of the aggregation coordinate ?

close

public void close()
           throws java.io.IOException

Release all resources associated with the aggregation

Throws:

java.io.IOException

persist

public void persist()
             throws java.io.IOException

Overriden in AggregationExisting

Throws:

java.io.IOException

persistRead

protected void persistRead()

finish

public void finish(CancelTask cancelTask)
            throws java.io.IOException

Throws:

java.io.IOException

buildDataset

protected abstract void buildDataset(boolean isNew,
                                     CancelTask cancelTask)
                              throws java.io.IOException

Throws:

java.io.IOException

buildCoords

protected void buildCoords(CancelTask cancelTask)
                    throws java.io.IOException

Throws:

java.io.IOException

syncExtend

public boolean syncExtend(boolean force)
                   throws java.io.IOException

Check to see if its time to rescan directory, and if so, rescan and extend dataset if needed.

Parameters:

force - if true, always rescan even if time not expired

Returns:

Throws:

java.io.IOException

syncDataset

protected void syncDataset(CancelTask cancelTask)
                    throws java.io.IOException

Throws:

java.io.IOException

sync

public boolean sync()
             throws java.io.IOException

Throws:

java.io.IOException

timeToRescan

protected boolean timeToRescan()

Rescan if recheckEvery time has passed

Returns:

if theres new datasets, put new datasets into nestedDatasets

Throws:

java.io.IOException

rescan

protected boolean rescan()
                  throws java.io.IOException

Throws:

java.io.IOException

getTypicalDataset

protected Aggregation.Dataset getTypicalDataset()
                                         throws java.io.IOException

Open one of the nested datasets as a template for the aggregation dataset.

Throws:

java.io.FileNotFoundException - if there are no datasets

java.io.IOException

makeProxies

protected void makeProxies(Aggregation.Dataset typicalDataset,
                           NetcdfDataset newds)
                    throws java.io.IOException

Throws:

java.io.IOException

read

public Array read(Variable mainv,
                  CancelTask cancelTask)
           throws java.io.IOException

Read an aggregation variable: A variable whose data spans multiple files.

Specified by:

read in interface ProxyReader

Parameters:

mainv - the aggregation variable

cancelTask - allow the user to cancel

Returns:

the data array

Throws:

java.io.IOException

readAggCoord

protected Array readAggCoord(Variable aggCoord,
                             CancelTask cancelTask)
                      throws java.io.IOException

Throws:

java.io.IOException

read

public Array read(Variable mainv,
                  CancelTask cancelTask,
                  java.util.List section)
           throws java.io.IOException,
                  InvalidRangeException

Read a section of an aggregation variable.

Specified by:

read in interface ProxyReader

Parameters:

mainv - the aggregation variable

cancelTask - allow the user to cancel

section - read just this section of the data, array of Range

Returns:

the data array section

Throws:

java.io.IOException

InvalidRangeException - if section is incorrect rank or shape.

readAggCoord

protected Array readAggCoord(Variable aggCoord,
                             CancelTask cancelTask,
                             java.util.List section)
                      throws java.io.IOException,
                             InvalidRangeException

Throws:

java.io.IOException

InvalidRangeException

scan

protected void scan(java.util.List<Aggregation.Dataset> result,
                    CancelTask cancelTask)
             throws java.io.IOException

Scan the directory(ies) and create nested Aggregation.Dataset objects. Directories are scanned recursively, by calling File.listFiles(). Sort by date if it exists, else filename.

Parameters:

result - add to this List objects of type Aggregation.Dataset

cancelTask - allow user to cancel

Throws:

java.io.IOException - if io error

makeDataset

protected Aggregation.Dataset makeDataset(java.lang.String cacheName,
                                          java.lang.String location,
                                          java.lang.String ncoordS,
                                          java.lang.String coordValueS,
                                          boolean enhance,
                                          NetcdfFileFactory reader)

Dataset factory, so subclasses can override

Parameters:

cacheName - a unique name to use for caching

location - attribute "location" on the netcdf element

ncoordS - attribute "ncoords" on the netcdf element

coordValueS - attribute "coordValue" on the netcdf element

enhance - open dataset in enhance mode

reader - factory for reading this netcdf dataset

Returns:

a Aggregation.Dataset