| • |
|
|
|
|||
|
||||
Q: I want to read data from a NetcdfFile and use another package to process the results, which doesnt use ucar.ma2.Array. How can I get the data out of the Array object most efficiently?
The most efficient way is to use this method on the Array class:
public java.lang.Object get1DJavaArray(java.lang.Class wantType);which will give back the Java primitive array, without copying if possible. For example if you have a type double Array object:
double[] ja = (double []) ma.get1DJavaArray( double.class);You then have to see if your chosen package has a constructor that can wrap the Java array without copying it. You will also need the multidimensional shape information: int[] shape = ma.getShape().
Q: How do I use NetcdfFile object caching?
Initialize the object cache by calling NetcdfDataset.initNetcdfFileCache(), then open files through NetcdfDataset.acquireFile() and NetcdfDataset.acquireDataset(). Note that you always close a file in the normal way, ie through NetcdfFile.close(). See the javadoc for those methods for more details.
The main reason to use object caching is in a high performance server application.
Q: How do I control where temporary files are placed (Disk Caching)?
See Disk Caching.
Q: I have lots of data in a proprietary format. How do I read it into the CDM?
You write a class that implements the IO Service Provider interface. See section 4 of the tutorial.
Q: I want to create a NetcdfFile from some other source than a file. But an IOSP is designed to get data from a RandomAccessFile. So what do I do?
An IOSP just has to satisfy the contract of IOServiceProvider. It doesnt matter how it gets satisfied. You can create an IOSP any way you want - its an interface. If you don't need the RandomAccessFile, you can ignore it. Use memory-resident data, a random-number generator, or any other way to satisfy a data request.
You can create a NetcdfFile with the protected constructor, and pass in your IOSP:
protected NetcdfFile(IOServiceProvider spi, RandomAccessFile raf, String location, CancelTask cancelTask);// need access to protected constructor class MyNetcdfFile extends NetcdfFile { MyNetcdfFile (IOServiceProvider spi, String name) { super(spi, null, name, null); } }Once you have a NetcdfFile, you can wrap it with a NetcdfDataset, a GridDataset, a PointFeatureDataset, etc, and use all the mechanism of subsetting already in those classes. Ultimately those call your IOSP for data, and you must return the data correctly, according to the interface contract.
Q: What about the "O" in IOServiceProvider? How does that work?
If you look at ucar.nc2.iosp.IOServiceProviderWriter, you can see the start of a possible standard mechanism for writing to different file formats. But it isnt used anywhere that would likely be useful to you. You probably just want to write your own class that takes a NetcdfFile object, and writes it to your file format, in whatever way suits you best. You might find the code at ucar.nc2.FileWriter useful to look at.
Q: How do I control the error messages coming out of the library?
The netCDF-Java library currently uses SLF4J logging. This alllows you to switch what logging implementation is used. See the Logging section of this page.
Q: Im using the log4j logging package. How do I get rid of the message "log4j:WARN No appenders could be found for logger (ucar.nc2.NetcdfFile). log4j:WARN Please initialize the log4j system properly" ?
Add the following to your startup code:
org.apache.log4j.BasicConfigurator.configure(); org.apache.log4j.Logger logger = org.apache.log4j.Logger.getRootLogger(); logger.setLevel(org.apache.log4j.Level.OFF)
Q: Is the Netcdf-Java library thread-safe?
Underneath a Variable/NetcdfFile is (usually) a java.io.RandomAccessFile object, which is not thread-safe, because it keeps the state of the file position. So even if all you want to do is read data in multiple threads, you need to synchronize, typically on the NetcdFile object. Alternatively, open a new NetcdfFile for each thread. The THREDDS Data Server (TDS) uses a cache of open NetcdfFile files by using the NetcdfDataset.acquireFile() method, which allows stateless handling of data requests minimizing file opening and closing.
Q: Do I need to synchronize if I use NetcdfDataset.acquireFile()?
The way that the cache is designed to work is that you get back a NetcdfFile object, which should then be used only in a single thread so that you dont need synchronization ("thread-confinement"), eg to answer a single request in a server. Until you release that NetcdfFile object, no one else can get it from the cache. If another request is made for that same NetcdfFile while its locked, a new NetcdfFile is opened. And of course, the cache itself is thread-safe. So if you use it properly, you never have to do synchronization yourself.
Q: The NcML in my TDS is not working. What should I do?
Generally its much easier to debug NcML outside of the TDS. Here are some guidelines on how to do that.
- Go to the TDS configuration catalog and extract the NcML:
- Find the problem dataset. Inside the <dataset> element will be a <netcdf> element, that is the NcML. Cut and paste into a file, say its called test.ncml (it must have suffix ncml or xml).
- Add the XML header to the top of it: <?xml version="1.0" encoding="UTF-8"?>
- Remove the recheckEvery attribute if present on the <scan> element.
- Make sure the referenced datasets are available. If its an aggregation, a simple thing to do is to copy two or more of the files and put them in the same directory as test.ncml. Use a scan element or explicitly list them in a <netcdf> element, with the location attribute being the reletive path name.
- Open test.ncml in the viewer tab of ToolsUI, to check for NcML errors. You now see directly what the modified dataset looks like. Modify test.ncml and re-open it until you get it right. The NcML tab allows you to edit and save the NcML file, but it is a very primitive editor.
- If its a grid dataset,open it in the FeatureTypes/Grid tab to make sure you see grids, to check for complete coordinate system. If you dont see the grids you expect, the CoordSys tab might be helpful. It takes some expertise to understand how Coordinate systems work. When all else fails, follow the CF specification.
- If its an aggregation, the NcML/Aggregation tab will show you the individual file in the aggregation.
- If its an FMRC aggregation, the Fmrc/FmrcImpl tab will show you the various datasets found.
- Onec things are working correctly, put your changes back into TDS catalog and restart the server
- Open your TDS catalog in the ToolsUI/THREDDS tab. Navigate to the dataset, and "open as file" or "open as dataset" (at bottom). You should see the same results as in steps 2 and 3.
Q: How do I work with unsigned integer types?
When accessing a file through the "raw interface", by opening the file through NetcdfFile.open() or NetcdfDataset.openFile(), you may see unsigned data types. These have the attribute _Unsigned = "true". See CDM Datatypes for more details on how to work with unsigned data arrays.
When accessing a file through the "dataset interface", by opening the file through NetcdfDataset.openDataset(), if there is a scale/offset attribute set on a Variable, the variable will be widended to the type of the scale/offset attribute, typically a float. In this case the unsigned data will be correctly handled, and doesnt need special handling.
Classic netCDF-3 format has only signed bytes. The CDM library often sees unsigned bytes coming from other data formats, and we made the decision not to automatically widen unsigned bytes to shorts. The data is delivered using Java integer types, which are signed, so its up to the application to check Variable.isUnsigned() and do the right thing when doing computations with the data. The library handles the conversion correctly when the scale/offset attributes are being used.
Q: Ok, so you read a lot of files, what about writing?
Netcdf-Java library has very limited support for writing files.. The only supported output format is the netCDF-3 "classic" file format using the classic data model. This means that you cannot write Groups, Structures, 64-bit integers, and other new data types with the netcCDF-Java library. See the ucar.nc2.NetcdfFileWriteable javadoc.
- ucar.nc2.FileWriter writes CDM files to netCDF-3 format. However, currently it does not try very hard to capture new features in this limited format.
- ucar.nc2.dataset.NetcdfDataset.main() is a cover for FileWriter, but you can also open NcML, OPeNDAP, and other datasets accessible with NetcdfDataset.openFile(). You can use it in command line mode like:
java -Xmx512m -classpath netcdfAll-4.1.jar ucar.nc2.dataset.NetcdfDataset -in <fileIn> -out <fileOut> [-isLargeFile]
- Use the latest version of the the netcdfAll jar
- -isLargeFile means to write a 64-bit offset netCDF file. You need version 4.1. or above for this feature.
- You can wrap the original dataset in NcML, to modify the file (for example use the <remove> element to remove the variables you dont want to save). The xml file must have the suffix ".ncml", and it becomes the "in" file:
java -Xmx512m -classpath netcdfAll-4.1.jar ucar.nc2.dataset.NetcdfDataset -in myfile.ncml -out out.nc
Q: What about writing NetCDF-4 format?
NetCDF-4 format is built on top of the HDF5 file format. It adds shared dimensions, so it is unfortunately not a strict subset of HDF5. HDF5 is a very complicated format, and at this time we do not have resources to write a 100% Java version for writing (we do have a 100% Java version for reading). We are considering a JNI interface to the NetCDF-4 C library, but there is no current timeline for that. So for now, you can only use the NetCDF-4 C library to write NetCDF-4 format files.
Q: Can I stream a NetcdfFile object to a client?
NetCDF is a random access format, so streaming is not always possible. The standard way to do this is to write to a disk file (so that you have random access), using ucar.nc2.FileWriter, then copy the file to the client. Note that you must stay within the "classic model" to write a netCDF-3 file.
There are experimental classes based on ucar.nc2.iosp.netcdf3.N3streamWriter, but these shouldnt be used in production without extensive testing. If you want to experiment, ucar.nc2.ft.point.writer.WriterCFPointObsDataset is an example class that uses stream writing. Stream writing is experimental and the APIs may change or be withdrawn in future releases.
Q: What kind of information should I put into my netCDF file to help others read it?
Thank you for asking, See:
- General Guildelines: http://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html
- Recommended Conventions: CF Conventions
- Proposed Conventions for Point Observations
This document is maintained by John Caron and was last updated on Oct 13, 2009
| Contact Us • Site Map • Search • Terms and Conditions • Privacy Policy • Participation Policy | |||||
|
|||||
