netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Hi Russ, > > >There are some advantages of sequence numbers over times: > > > - you don't have to worry about clock resolution and the possibility > > > that creation times of two objects are equal > > Hmm, we use the gettimeofday() routine, which returns values in > > microseconds, so this probably would not be too much of an issue, but I > > admit > > it certainly is possible. > > We ran into just this problem on a skiplist implementation (for LDM > not netCDF) that required a total ordering. Time stamps worked most > of the time, but if two events happened to get assigned the same > microsecond clock tick, we lost track of one of the corresponding > objects. On old machines, we never saw the problem, but it bit us > when we tried running on faster hardware. We ended up adding what was > essentially a sequence number to the timestamp to disambiguate > matching microsecond clock times. Well, I hope that we can create objects in the file fast enough that having only a microsecond resolution is a problem for HDF5 also... :-) > > Hmm, I think there may be some issues with a creation sequence number also: > > - The "last number issued" will need to be stored in the file (unlike > > creation times). > > - Should it be local to the group, or global to the file? There are > > pro's and con's to both: > > Global: > > - Pro: One number to track for file > > - Con: May have contention for updating this number in a > > parallel environment. > > - Con: Faster to roll over than a sequence number per group. > > - Con: Sequence numbers in one group will have gaps, if > > objects are created in other groups, which does not > > imply objects were deleted in the group. > > > > Local: > > - Pro: More consistent numbering within one group than a > > sequence number per file. > > - Con: May have contention for updating this number in a > > parallel environment. > > - Con: A new piece of metadata to update with every object > > created in a group. > > > > I guess I would tend toward a local (i.e. per group) sequence number. > > How's that sit with people? > > Good analysis of sequence number problems. I agree with you, local > seems to be adequate unless we chose to ignore Group semantics for the > netCDF-4 interface and just treated the Group name as part of a global > name for a netCDF-4 object. In that case, local would be a problem, > because two netCDF-4 objects that we wanted to iterate over in order > could get the same sequence number. Maybe this is an argument not to > treat Groups as just part of the name. Yes, local sequence numbers cut both ways sometimes... Since most (all?) current netCDF users should be used to a 'flat' file, putting all the objects in the root group of the file and using the creation order in that group seems like a reasonable default. Then, you could change the definition of the way the creation order information is used for netCDF 4 users so that the group structure was accounted for. BTW, I was looking through the netCDF 3 API for functions that take or return an 'index' in the file and I can't find one. Which function(s) applies to this situation? > For us, a different kind of local would also work: a set of sequence > numbers for Datasets, for each Dataset's Attributes, and for shared > dimension Scales. But if you have other uses for time stamps or > sequence numbers, our use shouldn't dictate the requirements, since > anything that allows us to determine the creation order of netCDF > variables, dimensions, and attributes would work. This is along lines that we've thought about for a long time: adding a live" index capability to HDF5 files, where every change to the file's metadata (object creation, modification, deletion and attribute creation & deletion) could update an index in the file in some way. I think this is a great idea, but I think it would be too much work at the current time. :-( Quincey