Hi Joe, > We have a lidar (HSRL) that records data in netCDF. We need to get the same > data copied (near real-time) to a server for archiving, and to allow > real-time access to the data files stored on the server. > > Currently, we use a Python script which watches the current netCDF file and > writes new records over a socket. As far as I know, this script parses the > netCDF file binary format. The files are recorded in netCDF3 classic format. > This script was written by a programmer at University of Wisconsin, where > the instrument was developed. > > Can you suggest a cleaner way of achieving this functionality? Depending on how much latency you can tolerate, you might use this special version of rsync for netCDF files developed by Joe Sirott and just invoke it every few seconds to mirror your data efficiently: http://www.epic.noaa.gov/epic/software/cdfsync/ The problem with that (and any other approach I know of that doesn't parse the netCDF file to be mirrored) is that it may open the data file to be read at an inopportune time, for example in the middle of writing a large record. In that case, the mirrored data might be inconsistent, because reading the last record might read an end-of-file before all the data is read. To guarantee that the input file is in a consistent state, I think the writing and copying programs both need to create/open the file with the NC_SHARE flag, which is designed so for one writer and multiple readers to access data concurrently. I'm not sure how else to do what you want. --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: KAE-542730 Department: Support netCDF Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.