Re: [bufrtables] More on table versions

Hi Milan and John,

Perhaps I can usefully contribute here.

We did originally have a checksum for each of the main sections of BUFR, but the justification was mainly to do with unreliable bit orientated telecoms, and limits on block sizes. We addressed those by saying that the telecoms protocol should handle them. We wanted a clean, orthogonal separation of transport protocol and content.

We never considered correcting errors between tables, messages and applications. We didn't even envisage more than one Master Table - multiple master tables were a bit of an afterthought, to accommodate the oceanographers. We never really thought through in detail how things would be administered. It was left as a trivial exercise for the readers!

From the outside, I can see you all adding bits here and there to fix
things. I would say Bufr is now in its 'baroque' or 'rococo' phase, and I would recommend some serious pruning/deprecation/simplification/re-factoring to get back to a more 'classical' structure, perhaps a 'BUFR- lite'. I think complexity, even if it is necessary, seems to be a barrier to wider take up of BUFR. I think the success of NetCDF and HDF came from simplicity and elegance first (and supplied software and APIs), and then complexity came after it had been taken up and the community recognised that the extras were needed.

I hope you can come up with a straightforward table maintenance process.

Best wishes, Chris

Chris Little  International Telecommunications
Met Office  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
Tel: +44(0)1392 886278 Fax: +44(0)1392 885681 Mobile: +44(0)7753 880514
E-mail: chris.little@xxxxxxxxxxxxxxxx

Met Office climate change predictions can be viewed on Google Earth
http://www.metoffice.gov.uk/research/hadleycentre/google/

-----Original Message-----
From: bufrtables-bounces@xxxxxxxxxxxxxxxx [mailto:bufrtables-bounces@xxxxxxxxxxxxxxxx] On Behalf Of John Caron
Sent: 18 August 2008 23:21
To: Milan.Dragosavac@xxxxxxxxx
Cc: bufrtables@xxxxxxxxxxxxxxxx
Subject: Re: [bufrtables] More on table versions

Hi Milan:

Thanks for your thoughts on this. One of the interesting questions is whether software can automatically detect if the wrong table is being used to decode. When the bit widths are wrong, the answer is usually yes, since one can "count bits" and compare to the data section length.

However, if scale/offset/units is wrong, then one must actually understand the data in order to detect problems, and even then it way not get detected.

Something that perhaps you have thought of before is to embed something like an MD5 checksum into the BUFR message, to be compared against the MD5 of the master table stored at the WMO. Obviously a lot of trouble, and even then not foolproof, but perhaps worth considering, especially for archived data.

Regards,
John

Milan Dragosavac wrote:
Hi John,

I believe you might find almost any master table version number in the
data. In practice in the section 1 you will find local version numbers

use as well although the local entries are not really used. In any
case you can always first try making link to version 13 and find out
how it goes. I am sure it will work in almost all cases. I will try to
address this problem at WMO ET/DRC Meetings.

Best regards
Milan

Milan Dragosavac
ECMWF
Shinfield Park, Reading, Berkshire, RG2 9AX, UK

Tel: (+44 118) 949 9403
Fax: (+44 118) 986 9450
Telex: 847908 ECMWF G
E-mail: milan.dragosavac@xxxxxxxxx

_______________________________________________
bufrtables mailing list
bufrtables@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/