netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Hi, I'm forwarding this summary of the Langley meeting on the "National Grid Project" that occurred during our workshop, for background information. --Russ Introduction: On May 1, 1992 there was a meeting at NASA Langley to discuss possible choices of a library and application programming interface for reading/writing/exchanging scientific data for use in the National Grid Project at MSU. In this case, the area of interest was mesh information and variables on a mesh. Another purpose of this meeting was to participate in and contribute to the ongoing efforts to establish a defacto scientific database library. The people attending were: Stewart Brown LLNL - PDBLib Linnea Cook LLNL Mike Folk NCSA - netCDF/HDF merge Adam Gaither NGP Chris Houck NCSA - netCDF/HDF merge Robert Jackson NARL Jeff Long LLNL - SILO Michael McLay NIST Bob Weston NASA Langley I (Linnea Cook) agreed to write up the results of this meeting. The results and action items are in the section with that title which follows later. I have also included the following background section to define all the acronyms used and to give some context for the decisions which were made. Most of this background information was covered during the Friday meeting at Langley. Background: For some time there has been interest and work in the scientific community for having a standard library and application programming interface for reading/ writing/exchanging scientific data. Three established defacto standards for this are: o HDF (Hierarchical Data Format) from NCSA (National Center for Supercomputer Applications), o netCDF (network Common Data Form) from Unidata Program Center and o CDF (Common Data Form) from NASA Goddard. CDF and netCDF are very similar; in fact, netCDF is a spin off from CDF. Several recent important steps have occurred to further the establishment of a single, more widely used I/O library for scientific data. NSF has recently decided to fund NCSA to put the netCDF interface on top of its HDF library and to convert all of their public domain tools (Image, Layout, etc.) to use the netCDF interface. NCSA is in touch with Unidata in regards to this merge project. This should help to unify two widely used standards. The Earth Observing Systems project (EOS) is expected to receive $3 billion in funding over the next decade. The Earth Observing System Data Information System (EOSDIS) part of the EOS project has selected NCSA's netCDF/HDF merge product as its scientific data I/O library. NCSA does not know how many people currently use HDF. However, when the latest version of HDF was released to its users, 2500 people downloaded a copy of this library within the first month. As part of this netCDF/HDF merge project, Mike Folk (who is in charge of HDF and the netCDF/HDF merge project) is also interested in some work which has been done by Jeff Long at Lawrence Livermore National Laboratory (LLNL) on the SILO library. SILO is a library which implements an application program interface for reading and writing scientific data. SILO uses the calling sequence of the netCDF library but has made two extensions to the netCDF interface - objects and directories. Objects allow a set of related variables and other data to be grouped together. Directories in SILO allow the user to structure a database into a hierarchy that is analogous to a UNIX file system. It is these two extensions to netCDF (objects and directories) which NCSA is interested in including in its netCDF/HDF merge. At a meeting between NCSA and LLNL it was determined that the SILO extensions appear to be very compatible with the NCSA merge of netCDF and HDF. NCSA wants to add these extensions and will do so provided their user community approves and provided they have funding to do this work. NCSA estimated that it will take them one month to finish the prototype version of the netCDF/HDF merge work. They would want to allow six months to do the SILO extensions once this work is started. Russ Rew (who leads the netCDF project at Unidata) and Jeff Long (the author of SILO) are currently corresponding to refine the SILO extensions to netCDF. They hope to agree upon these extensions and cooperate with NCSA and Unidata so that the same extensions are put into both the netCDF/HDF merge and into netCDF. Two other topics were mentioned but not resolved at the NCSA / LLNL meeting. These topics were the `standard' definition of some objects and the use of a socket library interface for reading data across a network. The SILO object extension (by itself) allows users to define their own objects but does not assign meaning to the objects. SLIDE (a companion library to SILO) has, however, defined objects for mesh data commonly found in physics simulations. An example is a `quadmesh' (quadrilateral mesh) - this object must include the dimension and coordinate data and also typically includes the mesh's labelling and unit information. NCSA and LLNL thought it was desirable to use these mesh object definitions as a starting point for the scientific community to define `standard' mesh objects for use in the netCDF/HDF merge product. However, since NCSA is not yet funded to do the two primitive extensions (objects and directories) it is somewhat premature to plan this. HDF currently has a socket library interface for reading HDF data across a network. SILO has a similar interface for reading SILO objects across a network. In both cases the code which uses HDF or SILO does not know whether the data is coming from a disk file or a network connection. NCSA and LLNL think it may be possible to combine the two socket libraries some time in the future since each addresses a different data type but that it was premature to evaluate this now. The PDBLib (Portable Database Library) scientific database library is of considerable interest to the National Grid Project because of its speed and flexibility. PDBLib is similar to HDF in that both the library and the file it produces are portable. One major difference between PDBLib and HDF is that PDBLib allows the user to define C-like structures, then read and write these structures in one operation. The structures can contain primitive data, pointers to primitive data, other structures, and pointers to other structures. PDBLib also has a more general conversion model - it can write in native format, then read that on any other machine. Or, it can create a file on one machine in any other machine's format. HDF can read/write data in a machine's native format but can not move this file to any other machine which uses a different format. HDF also can read/write IEEE format on any machine - this IEEE format file is portable to any computer. PDBLib was developed at LLNL by Stewart Brown. The SILO interface is currently implemented on top of PDBLib. Results and Action Items from the May 1 Meeting with the National Grid Project: The following consensus was achieved during this meeting: 1. The National Grid Project would like to use the netCDF/HDF merge product being developed at NCSA and is also interested in the SILO extensions and SLIDE object definitions. 2. The attendees agreed to send Mike Folk requirements for HDF - some of this will be based on experience with the current HDF library. Also, there was concern that there is functionality in PDBLib which is not in HDF. Stewart Brown will be sending out the PDBLib manual to interested parties. Those not familiar with PDBLib said they would examine its functionality and provide Mike with feedback on what they saw as additional capabilities which they would like from the PDBLib capabilities. All of this should be written input. 3. Since NCSA is interested in adding the SILO extensions to their netCDF/HDF merge product, the people at this meeting agreed to examine the SILO extensions and get back to Mike Folk with their feedback on these extensions (opinions, changes, additions). These two SILO extensions are the directory and object primitives. This should be written input. 4. Mike Folk will send the SILO extensions out to his user community for feedback. 5. Part of the SILO document includes the definition of higher level mesh objects on top of the netCDF interface (with the two SILO primitives). We agreed to look at these objects as a possible starting point for defining 'standard' mesh objects for use across many sites. 6. Mike Folk agreed to send us copies of all written input sent to him by this group to us (with possibly some editing). He also agreed to put together a list of additional requirements for the netCDF/HDF merge product and send them to us. 7. The next step which was mentioned was that NCSA would need funding to do additional work beyond the netCDF/HDF merge. 8. The netCDF/HDF merge product will be available as a beta product by the end of July and as a fully released product by May 1993. The National Grid Project will need a library sooner than that. The SILO/netCDF interface on top of PDBLib may be used as an interim solution allowing the netCDF/HDF merge library to replace it with little or no change to the application programming interface. Other Information: One desire expressed at this meeting was for a way of writing data to disk without going through the translation to IEEE floating point format and being able to read this data and translate it later, if necessary. Another desire was to be able to write a code's internal data structures directly to disk. Some subsequent discussion indicated that being able to write data quickly and with little overhead (little extra information written to disk) was the basic requirement. Another part of this requirement seems to be the ability to write any data to disk without first getting an 'approved' tag or data type implemented. This was for use during the development stage. All agreed that eventually the tags would be officially requested, granted and documented. Since the issue of writing a code's internal data structures directly to disk received considerable comment, we should specifically address this in our feedback to Mike Folk. Related to this is the question of whether it is important to be able to use other tools (such as graphics codes) to read (and display or do other operations on) this data? Adam Gaither from NGP volunteered to be a contact point for disseminating information to this group. NGP is another good forum for pushing a standard scientific database library and Adam wants NGP to help with and participate in this process. Michael McKay from NIST talked about several other related standards in the scientific community (STEP, IGES, OMG, OSI). He will be forwarding information on relevant standards to the attendees. Unless requested otherwise, all attendees were added to NCSA's mailing list for the discussion of the netCDF/HDF merge project. Send mail to hdf-netcdf- request@xxxxxxxxxxxxx if you wish to be removed. If you have any questions, corrections or additions, please call me at the phone listed below. Or you can send email to me though Jeff Long. I will send out my own email address as soon as I have one. My name and address were left off the list which Adam Gaither sent out, so if you would please add my name and address to your copy of the list I would appreciate it. -------------------------------------- Linnea Cook Lawrence Livermore National Laboratory B Group Leader P. O. Box 808, L-35 Livermore, California 94550 Desk: 510-422-1686 FAX: 510-422-3389 E-Mail: (you can reach me through Jeff Long's E-Mail) jwlong@xxxxxxxx