On Thu, Dec 13, 2007 at 10:14:04AM -0600, Rick Hedin wrote:
> Greetings!
>
> I am making an information collector that will collect information from
> a bunch of processes on a bunch of machines. Initially, each process
> will write to it's own file on it's machine. Eventually, each process
> will contribute to a collector process on one machine using UDP.
Have you considered using MPI for the data exchange? Might be easier
and more portable than UDP.
> I'd like to use netCDF because of its self-describing nature, and
> because the information is uniform across machine architectures. But I
> wonder if netCDF can interpret a record in isolation, or if it depends
> on an overall file structure. I will end up with one big file
> containing records from a bunch of different sources.
>
> If a netCDF file is just a few records of common header information at
> the top, followed by a bunch of records, each of which is in an allowed
> format, but which are not in any particular order, I don't anticipate
> any problems.
>
> Has anybody collected records from a bunch of sources and put them all
> together in one file?
This is the usual mode of operation for a parallel (MPI using)
application. I think it's much less common if you have serial netcdf:
usually in serial netcdf a master process coordinates and controlls
all access to the netcdf dataset.
If your application makes sense for MPI (and it might not..), you
might want to use parallel-netcdf (www.mcs.anl.gov/parallel-netcdf) to
create the dataset. you'll still be able to use the whole ecosystem
of netcdf viewers.
==rob
--
Rob Latham
Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA B29D F333 664A 4280 315B
