Re: [bufrtables] More on table versions

  • Subject: Re: [bufrtables] More on table versions
  • From: John Caron <caron@xxxxxxxxxxxxxxxx>
  • Date: Sat, 23 Aug 2008 11:02:57 -0600
Hi Chris:

Very good points about data models/formats becoming complex. There are a number of features that appear to complicate things without obvious corresponding benefit. Perhaps it would be helpful if we start to enumerate exactly what are the basic features and which not?
As I mentioned in a previous post, the checksum idea was to verify which table 
was used, rather than message integrity, which can now be assumed to be handled 
at a lower layer. Whether that's an idea whose complexity is worth it is 
another issue!

Some bufr complexity appears to be motivated by keeping the record small. I 
wonder whether a simpler data layout with dictionary based compression (eg LZW) 
might not work better, now that CPUs are so much faster than IO. I will be 
doing some testing with that idea, once my decoder is reliably working.

Regards,
John

Little, Chris wrote:
Hi Milan and John,

Perhaps I can usefully contribute here.

We did originally have a checksum for each of the main sections of BUFR, but 
the justification was mainly to do with unreliable bit orientated telecoms, and 
limits on block sizes. We addressed those by saying that the telecoms protocol 
should handle them. We wanted a clean, orthogonal separation of transport 
protocol and content.

We never considered correcting errors between tables, messages and 
applications. We didn't even envisage more than one Master Table - multiple 
master tables were a bit of an afterthought, to accommodate the oceanographers. 
We never really thought through in detail how things would be administered. It 
was left as a trivial exercise for the readers!

From the outside, I can see you all adding bits here and there to fix things. I would say Bufr is now in its 'baroque' or 'rococo' phase, and I would recommend some serious pruning/deprecation/simplification/re-factoring to get back to a more 'classical' structure, perhaps a 'BUFR- lite'. I think complexity, even if it is necessary, seems to be a barrier to wider take up of BUFR. I think the success of NetCDF and HDF came from simplicity and elegance first (and supplied software and APIs), and then complexity came after it had been taken up and the community recognised that the extras were needed.

I hope you can come up with a straightforward table maintenance process.
Best wishes, Chris

Chris Little  International Telecommunications
Met Office  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
Tel: +44(0)1392 886278  Fax: +44(0)1392 885681  Mobile: +44(0)7753 880514
E-mail: chris.little@xxxxxxxxxxxxxxxx

Met Office climate change predictions can be viewed on Google Earth
http://www.metoffice.gov.uk/research/hadleycentre/google/