Benno, > MODIS is kept in 317 tiles per timestep, so to make 1 global image, I need > to read 1 tile each from 317 different files, a common structure for GIS > community data (there are 8 or so tiles per file corresponding to different > variables). > netcdf4 is reading all the metadata on open, which means in this case that > all the disk blocks are touched (my guess is that at least some of the > metadata is spread amongst the data blocks). I am not using the metadata at > all (they are tiles, when I read the tiles I already know the structure from > a previous analysis/read of the metadata), so this is a bit of a waste, and > very slow if the metadata is spread throughout the file, as it seems to be. I'm surprised that every disk block is read. That's not typical for netCDF-4 files that use HDF5 as a storage layer. Typically there is only a small amount of metadata, which comprises - the name and size of each dimension - the name, type, and values for each attribute - the name, type, and shape for each variable - the association information linking variables and attributes - other variable properties, such as compression level and layout - information about group names and group links - information about definitions of user-defined types - the B-trees of chunks for each chunked variable Although this information is scattered around the file, it doesn't involve every disk block. Each variable chunk is typically one or more disk blocks that are entirely data, perhaps compressed. Compressed data is not uncompressed until its read, and reading a single tile should not access any of the other tiles, if a tile is one or more chunks. > Since I am reading the entire tile once for a given variable (just not all > the different variables), chunking within the file does not really matter, > and reuse does not happen. Right, chunking is irrelevant in that case. > Aggregation like this is a common use case for netcdf -- not necessarily > common yet for tiles, but certainly common in time. So would you consider > improving the performance in this case where the metadata is not read for > use along with the data? I've discussed this previously with Ed Hartnett, who implemented the HDF5 reading code. It apparently would require some extensive changes to the current code. However, I've created a Jira issue for this, so it's on our list to investigate, and you can follow or comment on the issue here: https://www.unidata.ucar.edu/jira/browse/NCF-132 For now, you would probably be better off reading the MODIS data through the HDF5 library than through the netCDF API. If you try that, I'd be interested in how much the performance improves. --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: IDT-559068 Department: Support netCDF Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.