netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Ed, > Russ and John, are we going to say that not all types are convertible > in netCDF? At this moment, any type can be converted into any other, > but are we going to try that with string? Doesn't seem to make much > sense... No, in netCDF there is no conversion supported between text and numeric types. As it says in section 3.3 of the Users Guide: If the netCDF external type for a variable is char, only character data representing text strings can be written to or read from the variable. No automatic conversion of text data to a different representation is supported. The only conversions supported are among numeric types. --Russ >From owner-netcdf-hdf@xxxxxxxxxxxxxxxx 16 2004 Jul -0600 10:09:36 Message-ID: <wrxu0w8q6gv.fsf@xxxxxxxxxxxxxxxxxxxxxxx> Date: 16 Jul 2004 10:09:36 -0600 From: Ed Hartnett <ed@xxxxxxxxxxxxxxxx> In-Reply-To: <200407160427.i6G4RoTQ005204@xxxxxxxxxxxxxxxxxxxxxx> To: netcdf-hdf@xxxxxxxxxxxxxxxx Subject: Re: parallel I/O and netCDF-4 Received: (from majordo@localhost) by unidata.ucar.edu (UCAR/Unidata) id i6GG9bt9007626 for netcdf-hdf-out; Fri, 16 Jul 2004 10:09:37 -0600 (MDT) Received: from rodney.unidata.ucar.edu (rodney.unidata.ucar.edu [188.8.131.52]) by unidata.ucar.edu (UCAR/Unidata) with ESMTP id i6GG9aaW007622 for <netcdf-hdf@xxxxxxxxxxxxxxxx>; Fri, 16 Jul 2004 10:09:36 -0600 (MDT) Organization: UCAR/Unidata Keywords: 200407161609.i6GG9aaW007622 References: <200407160427.i6G4RoTQ005204@xxxxxxxxxxxxxxxxxxxxxx> Lines: 153 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-netcdf-hdf@xxxxxxxxxxxxxxxx Precedence: bulk Reply-To: netcdf-hdf@xxxxxxxxxxxxxxxx Quincey Koziol <koziol@xxxxxxxxxxxxx> writes: > From HDF5's perspective, you have to use H5Pset_fapl_<foo>(params) to > choose to use a particular file driver to access a file. Probably something > like this should be exported/translated out to the netCDF4 layer for users to > choose which driver to access the file with. > Here's the URL for the parallel HDF5 info currently: > http://hdf.ncsa.uiuc.edu/HDF5/PHDF5/ I'm seeing three steps to parallel HDF5: 1 - Initialize MPI 2 - When opening/creating the file, set a property in file access properties. 3 - Every time reading or writing file, pass a correctly set transfer property. Does that seem to sum it up? But I see below that you are also asking that "these properties must be set to the same values when they > are used in a parallel program," What do you mean by that? In parallel I/O do multiple processes try and create the file? Or does one create it, and the rest just open it? Sorry if that seems like a dumb question! > > > For reading, what does this mean to the API, if anything? > Well, I've appended a list of HDF5 API functions that are required to be > performed collectively to the bottom of this document (I can't find the link > on our web-pages). > > > Everyone gets to open the file read-only, and read from it to their > > heart's content, confident that they are getting the most recent data > > at that moment. That requires no API changes. > > > > Is that it for readers? Or do they get some special additional > > features, like notification of data arrival, etc? > User's would also need the option to choose to use collective or > independent I/O when reading or writing data to the file. That reminds me - > are y'all planning on adding any wrappers to the H5P* routines in HDF5 which > set/get various properties for objects? This is truly an important question that I will treat in it's own email thread... > > Quincey > > ============================================================== > > Collective functions: > H5Aclose (2) > H5Acreate > H5Adelete > H5Aiterate > H5Aopen_idx > H5Aopen_name > H5Aread (6) > H5Arename (A) > H5Awrite (3) > > H5Dclose (2) > H5Dcreate > H5Dfill (6) (A) > H5Dopen > H5Dextend (5) > H5Dset_extent (5) (A) > > H5Fclose (1) > H5Fcreate > H5Fflush > H5Fmount > H5Fopen > H5Funmount > > H5Gclose (2) > H5Gcreate > H5Giterate > H5Glink > H5Glink2 (A) > H5Gmove > H5Gmove2 (A) > H5Gopen > H5Gset_comment > H5Gunlink > > H5Idec_ref (7) (A) > H5Iget_file_id (B) > H5Iinc_ref (7) (A) > > H5Pget_fill_value (6) > > H5Rcreate > H5Rdereference > > H5Tclose (4) > H5Tcommit > H5Topen > > Additionally, these properties must be set to the same values when they > are used in a parallel program: > File Creation Properties: > H5Pset_userblock > H5Pset_sizes > H5Pset_sym_k > H5Pset_istore_k > > File Access Properties: > H5Pset_fapl_mpio > H5Pset_meta_block_size > H5Pset_small_data_block_size > H5Pset_alignment > H5Pset_cache > H5Pset_gc_references > > Dataset Creation Properties: > H5Pset_layout > H5Pset_chunk > H5Pset_fill_value > H5Pset_deflate > H5Pset_shuffle > > Dataset Access Properties: > H5Pset_buffer > H5Pset_preserve > H5Pset_hyper_cache > H5Pset_btree_ratios > H5Pset_dxpl_mpio > > Notes: > (1) - All the processes must participate only if this is the last > reference to the file ID. > (2) - All the processes must participate only if all the file IDs for > a file have been closed and this is the last outstanding object > ID. > (3) - Because the raw data for an attribute is cached locally, all > processes must participate in order to guarantee that future > H5Aread calls return the correct results on all processes. > (4) - All processes must participate only if the datatype is for a > committed datatype, all the file IDs for the file have been closed > and this is the last outstanding object ID. > (5) - All processes must participate only if the number of chunks in > the dataset actually changes. > (6) - All processes must participate only if the datatype of the > attribute a a variable-length datatype (sequence or string). > (7) - This function may be called independently if the object ID does > not refer to an object that was collectively opened. > > (A) - Available only in v1.6 or later versions of the library. > (B) - Available only in v1.7 or later versions of the library. >From owner-netcdf-hdf@xxxxxxxxxxxxxxxx 16 2004 Jul -0600 10:16:18 Message-ID: <wrxpt6wq65p.fsf@xxxxxxxxxxxxxxxxxxxxxxx> Date: 16 Jul 2004 10:16:18 -0600 From: Ed Hartnett <ed@xxxxxxxxxxxxxxxx> In-Reply-To: <200407160427.i6G4RoTQ005204@xxxxxxxxxxxxxxxxxxxxxx> To: netcdf-hdf@xxxxxxxxxxxxxxxx Subject: wrap HDF5 property lists for netCDF-4? Received: (from majordo@localhost) by unidata.ucar.edu (UCAR/Unidata) id i6GGGJ4r008530 for netcdf-hdf-out; Fri, 16 Jul 2004 10:16:19 -0600 (MDT) Received: from rodney.unidata.ucar.edu (rodney.unidata.ucar.edu [184.108.40.206]) by unidata.ucar.edu (UCAR/Unidata) with ESMTP id i6GGGIaW008526 for <netcdf-hdf@xxxxxxxxxxxxxxxx>; Fri, 16 Jul 2004 10:16:18 -0600 (MDT) Organization: UCAR/Unidata Keywords: 200407161616.i6GGGIaW008526 References: <200407160427.i6G4RoTQ005204@xxxxxxxxxxxxxxxxxxxxxx> Lines: 34 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-netcdf-hdf@xxxxxxxxxxxxxxxx Precedence: bulk Reply-To: netcdf-hdf@xxxxxxxxxxxxxxxx Quincey raises an interesting question: are we going to somehow wrap HDF5 property lists? Or shall we expose them to the netCDF-4 user? For example, the HDF5 file creation process involves two property lists, one for creation properties and one for access properties. Now we currently make various assumptions about what is desired when handling a nc_create call. That is, we just go with default property lists. But we probably also want to add a function to netcdf which directly exposes those property lists, allowing the user to construct and manipulate them with HDF5 functions, and then taking the property lists and handing them down to the HDF5 file creation function that gets called. This is a bit of a weird mix of HDF5 and netCDF-4, but it will allow the user to take advantage of new features in HDF5 without updating any netCDF-4 code. Or perhaps this is a bridge too far for Russ and Mike in terms of mixing the interface. I note that we face a similar decision about compound types. We could wrap HDF5 functions in netCDF functions that conform with our current interface, but why not just use the perfectly good HDF5 interface for creating compound types, and then taking the hid_t into an netCDF function? (Of course then we have to ask: what if they use a HDF5 type not supported in netCDF, if there are any such types?) Ed