Thanks Bryan. Hope you don't mind if I cc to the lists. Seems that
it should go on the record.
Platform dependent ... which is exactly what Steve Emmerson was
telling me (20040213: gdbm file reader program).
However, maybe Peter Neilley or others could chime in on his comments
in his README.linux file in the weather v4.10 distribution:
+++++
WEATHER for LINUX is a fully functional implementation. All aspects of
WEATHER should work. A fair amount of effort went into making WEATHER
work on a PC (Pentium) platform when the database is created or resides
on a UNIX machine of a different architucture. This is only an issue
for binary data files (gdbm metar datafiles, profiler data, and GEMPAK
datafiles) since PC's and most other common UNIX machines represent a
binary integer in opposite byte order. Anyway, you should find all
aspects of WEATHER functional. Report anything you find to the
contrary.
+++++
I can report that the gempak model files are processable by the weather
program using gempak libraries built on the relevant architecture
(PC/linux,SGI/Irix). Perhaps that verifies the byte-swap fix. But it
suggests
that he'd solved the gdbm off_t issue, which I guess you would seriously
doubt. But could it be that the off_t is not an issue with gdbm v1.7.3?
Robb Kambic reminds that trying to use gdbm v1.8 is a common
mistake but doesn't mention the offset issue as a specific problem.
If it's a general gdbm problem, this leaves me with a quandry which
perhaps has been solved by others -- maybe with a decoder, a different
pqsurf/pqact action, or a patch to weather: how to extract only the one
metar report for a station, preferably the one with REMARKS, via 'FLAT'
metar from the ascii decoded metar reports since that's the only format
universal to the architectures. [The -nosub weather option only filters
duplicates by exact check sum on the whole report, which won't do the
job.]
Any suggestions?
-Neil
On Friday, February 13, 2004, at 07:39 PM, Bryan C. Hahn wrote:
On Fri, 13 Feb 2004, Neil R. Smith wrote:
Anyone experiencing difficulties with P.Neilley's weather program
built on linux (say, Red Hat) reading gdbm metar files generated by
ldm DBFILE action (pqsurf) on another platform/architecture?
Should endian-ness be an issue? Peter's comments in the source code
file README.linux (ver 4.10) suggests that it shouldn't be an issue.
Neil,
I do know that gdbm files are NOT platform independent (at least as of
gdbm version 1.8). I have personal experience with gdbm files on HPUX
10
(big endian), FreeBSD, and Linux (little endian). No gdbm file I've
ever
created on one platform has been readable by software on another.
One specific problem is the storage of offsets in the file. The gdbm
library uses system-defined types for the specification of these
offsets.
Here's the definition of the header:
typedef struct {
int header_magic; /* 0x13579ace to make sure the header is
good. */
int block_size; /* The optimal i/o blocksize from stat.
*/
off_t dir; /* File address of hash directory table.
*/
int dir_size; /* Size in bytes of the table. */
int dir_bits; /* The number of address bits used in the
table.*/
int bucket_size; /* Size in bytes of a hash bucket struct.
*/
int bucket_elems; /* Number of elements in a hash bucket. */
off_t next_block; /* The next unallocated block address. */
avail_block avail; /* This must be last because of the psuedo
array in avail. This avail grows to
fill
the entire block. */
} gdbm_file_header;
The off_t type is system dependent.
On FreeBSD (4.4 and later):
sizeof ( off_t ) is 8
On Linux (Redhat 7.3):
sizeof ( off_t ) is 4
So that's an immediate problem.
Endian-ness is an even bigger obstacle.
I'm surprised (and a bit disappointed) that the gdbm developers early
on
didn't pay strict attention to developing a format which was
platform-independent. This problem has caused us to abandon this
otherwise useful data storage mechanism in most cases. We're using
Berkeley DB in most of the replaced cases.
- Bryan
-Neil
--
Neil R. Smith, Comp. Sys. Mngr. neils@xxxxxxxx
Dept. Atmospheric Sci., Texas A&M Univ. 979/845-6272 FAX:979/862-4466
--
Neil R. Smith neils@xxxxxxxxxxxxxxxxxx
Comp.Sys.Mngr. (979)845-6272
Dept. Atmospheric Sciences/Texas A&M University
