Re: [bufrtables] More on table versions

NOTE: The bufrtables mailing list is no longer active. The list archives are made available for historical reasons.

Hello Everyone,
I'm now back from a restful vacation and can add my proverbial $0.02 to the discussion.
First of all, Stan is correct that scale, reference and bit width values should never change between table versions. Once a descriptor has been included in an operational version it is considered static. This means that you theoretically only ever need the latest version of Table B in order to be able to decode everything, and any discrepancies for a particular descriptor between successive table versions are indeed typos.
Having said that, I do believe it is useful to maintain previous table versions, because they do allow you to go back and determine when certain descriptors became available for use. For example, if you received a message that said it was using version 8 of the tables, but it contained a particular descriptor that only became available beginning with version 10, having all of the previous versions available would allow you to easily diagnose such an error.
Please note that I'm only sharing with you what is the current WMO practice, and I'm not making any personal statement of opinion one way or the other as to whether this is a good thing. While the current practice, whereby each successive Table B is always a superset of all the previous versions, does simplify the task of decoding, it also has caused some of the descriptor classes to fill up much more quickly than would otherwise be necessary. A good example is Class 12, where we initially had a descriptor for every conceivable type of temperature value, but with scale 1 which only allowed accuracy to one digit beyond the decimal. After these were in use for a while, it was discovered that we had a problem, because most instruments report temperatures in Celsius whereas BUFR uses the SI unit of Kelvin, but the conversion factor between Celsius and Kelvin is 273.15, and different architectures didn't always handle the rounding the same way. This meant you could have one center take an observation in Celsius, then convert it to Kelvin for encoding into BUFR, then send that BUFR message to a different center, and the second center could, when converting back to Celsius, obtain a value that was off by as much as two-tenths of a degree from the original observed value. With hindsight, we realized that the best solution was to just store all temperatures using scale 2, but the practice was (and still is!) to never modify the characteristics of an existing descriptor once it has been in the table. So we ended up creating a second copy of each existing temperature descriptor, each with it's own new reference number and a scale factor of 2, when an alternative solution could have been to simply increase the scale of each existing temperature descriptor effective with the next version of the tables, and then rely on everyone's encoder/decoder software to differentiate properly between the old and new versions of each descriptor based on the table version number encoded within the message.
I realize that was a somewhat long-winded historical example, but an important point is that WMO could possibly change their practice at some point in the future (especially as some of the Table B classes are now close to completely full!), and this is another reason why it would be a good idea for us to hold on to all previous versions of these tables, even though right now it's not technically necessary. The easiest way to do this would be to include the version number somewhere in the filename within our collective archive.
As for BUFR edition numbers (about which I've also sensed some confusion in the ongoing email thread), these are a different animal entirely. Whereas table version number changes do not require any corresponding change to BUFR encoder/decoder software, the edition number is incremented only when changes are introduced which require corresponding changes to actual encoding and decoding software (for example, if new Table C operators are introduced, or the format of BUFR Section 1 is modified). In this case, the software needs to be able to simultaneously handle multiple editions (i.e. formats) of BUFR messages, which of course adds to programming complexity, and so such changes are made much less frequently and with much more advance notice. 

I hope this helps clarify any confusion!

Best regards,
-Jeff


John Caron wrote:

Hi Stan, et al:
I previously reported changes in various versions of the mel-bufr tables which led to thinking BUFR descriptors in the WMO master tables were changing between versions (previous posts copied below, for completeness). 
Now Im looking at the ECMWF tables used in the BUFRDC encoding/decoding 
software. They fail to confirm the differences in the mel-bufr table. This 
leads me to suspect the accuracy of the mel-bufr tables. So perhaps 
(hopefully), Stan, you are right that elements cant change once they become 
operational. Still, I wonder why mel-bufr and ECMWF keep versioned copies of 
the tables?

Also what to make of element 0-22-39 in the latest Table B document 
(http://www.wmo.int/pages/prog/www/WMOCodes/Operational/BUFR/BufrTabB-11-2007.pdf)
 which has changed the bit width to 13, when it has been 12 all along? Must be 
a typo; is anyone using that value(!) ?

ECMWF tables have a few changes of their own between versions. It could be that they are 
typos, that im looking at "pre-operational" values, or things that are local to 
ECMWF.
The lack of table management is quite a burden on developing a general reading package. 
To reiterate what we are thinking here at Unidata:

In the absence of a better mechanism, we will publish on the Unidata web site 
the BUFR tables we are using for our BUFR reading software, in various standard 
formats like XML and ASCII, in the hopes that any mistakes in them will be 
quickly caught and fixed. We will try to obtain canonical sources and have 
tools to compare representations to check for errors. We will encourage anyone 
else producing BUFR messages intended for data exchange (as opposed to internal 
consumption only) to send us their tables or links to their tables, which we 
will make available on our web site.

As soon as WMO or any other authoritative organization is willing and able to 
take over these tasks, and publish canonical machine-readable BUFR tables, we 
will point everyone to that site(s) instead.

John

PS: Please let me know if you'd like to be removed from this list, or go to 
http://www.unidata.ucar.edu/mailing_lists/


-----Original Message-----
From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx] Sent: 05 August 2008 20:03 
To: Kellett, Stanley
Cc: Wright, Bruce; Jeff Ator; Ross, Gil; Michelle Mainelli; Scott
Jacobs; Paul Hamer; Chris Macdermaid; Robb Kambic
Subject: Re: table versions

Hi Stan:

Thanks so much for your reply. I hope the flies arent biting as hard as
in Wyoming where I was camping last week.

If new versions of the WMO master table can only add new elements, this
would be very good news. It would mean that one really only needs the
latest version of the table. Is there somewhere that is explicitly
stated in the WMO BUFR specification?

The worry that I have is that if a mistake is made in a published
version of a table, and is used to code a BUFR message, that message
will be incorrectly read by a decoder using the corrected version of the
table. Theres no way to unambiguously know what table the coder used. 
We have been studying the differences between versions of the tables,
using the mel-bufr tables. While we are not sure of their exact
provenance, they seem to be widely used. 
Below is a summary of the differences in scale,offset and/or bit width
among mel-bufr versions 2 through 13. The difference is always with the
current version (13); "B3M-000-003-B.diff" for example means it is the
difference with mel-bufr table version 3. For example the line 
0-2-42
 width 6 != 2 (B2M-000-002-B.diff)

means that the bit width for field 0-2-42 was 6 for version 2, and 2 for
version 13.

--------------------------------

0-2-42
 width 6 != 2 (B2M-000-002-B.diff)
 width 6 != 2 (B3M-000-003-B.diff)
 width 6 != 2 (B3M-000-004-B.diff)
 width 6 != 2 (B3M-000-005-B.diff)

0-2-91
 scale 0 != 4 (B3M-000-008-B.diff)
 refer 4 != 0 (B3M-000-008-B.diff)
 scale 0 != 4 (B3M-000-009-B.diff)
 refer 4 != 0 (B3M-000-009-B.diff)
 scale 0 != 4 (B3M-000-010-B.diff)
 refer 4 != 0 (B3M-000-010-B.diff)

0-2-140
 scale 2 != 0 (B2M-000-002-B.diff)
 scale 2 != 0 (B3M-000-003-B.diff)
 scale 2 != 0 (B3M-000-004-B.diff)
 scale 2 != 0 (B3M-000-005-B.diff)

0-2-151
 width 4 != 11 (B3M-000-006-B.diff)
 width 4 != 11 (B3M-000-007-B.diff)
 width 4 != 11 (B3M-000-008-B.diff)
 width 4 != 11 (B3M-000-009-B.diff)

0-7-9
 refer 0 != -1000 (B3M-000-008-B.diff)

0-10-9
 refer 0 != -1000 (B3M-000-008-B.diff)

0-11-52
 width 14 != 13 (B3M-000-008-B.diff)

0-11-82
 width 13 != 14 (B3M-000-008-B.diff)

0-13-60
 refer -10 != -1 (B3M-000-008-B.diff)
 refer -10 != -1 (B3M-000-009-B.diff)
 refer -10 != -1 (B3M-000-010-B.diff)

0-21-85
 refer -3000 != 0 (B2M-000-002-B.diff)
 refer -3000 != 0 (B3M-000-003-B.diff)
 refer -3000 != 0 (B3M-000-004-B.diff)
 refer -3000 != 0 (B3M-000-005-B.diff)
 refer -3000 != 0 (B3M-000-006-B.diff)

0-22-39
 width 12 != 13 (B2M-000-002-B.diff)
 width 12 != 13 (B3M-000-003-B.diff)
 width 12 != 13 (B3M-000-004-B.diff)
 width 12 != 13 (B3M-000-005-B.diff)
 width 12 != 13 (B3M-000-006-B.diff)
 width 12 != 13 (B3M-000-007-B.diff)
 width 12 != 13 (B3M-000-008-B.diff)
 width 12 != 13 (B3M-000-009-B.diff)
 width 12 != 13 (B3M-000-010-B.diff)
 width 12 != 13 (B3M-000-011-B.diff)
 width 12 != 13 (B3M-000-012-B.diff)

0-24-1
 scale -1 != -11 (B2M-000-002-B.diff)
 scale -1 != -11 (B3M-000-003-B.diff)
 scale -1 != -11 (B3M-000-004-B.diff)
 scale -1 != -11 (B3M-000-005-B.diff)

0-24-2
 scale -1 != -11 (B2M-000-002-B.diff)
 scale -1 != -11 (B3M-000-003-B.diff)
 scale -1 != -11 (B3M-000-004-B.diff)
 scale -1 != -11 (B3M-000-005-B.diff)

0-25-50
 scale 0 != 4 (B2M-000-002-B.diff)
 refer 0 != -131072 (B2M-000-002-B.diff)  width 7 != 18
(B2M-000-002-B.diff)  scale 0 != 4 (B3M-000-003-B.diff)  refer 0 !=
-131072 (B3M-000-003-B.diff)  width 7 != 18 (B3M-000-003-B.diff)  scale
0 != 4 (B3M-000-004-B.diff)  refer 0 != -131072 (B3M-000-004-B.diff)
width 7 != 18 (B3M-000-004-B.diff)  scale 0 != 4 (B3M-000-005-B.diff)
refer 0 != -131072 (B3M-000-005-B.diff)  width 7 != 18
(B3M-000-005-B.diff)  scale 0 != 4 (B3M-000-006-B.diff)  refer 0 !=
-131072 (B3M-000-006-B.diff)  width 7 != 18 (B3M-000-006-B.diff)

0-25-51
 width 12 != 7 (B2M-000-002-B.diff)
 width 12 != 7 (B3M-000-003-B.diff)
 width 12 != 7 (B3M-000-004-B.diff)
 width 12 != 7 (B3M-000-005-B.diff)
 width 12 != 7 (B3M-000-006-B.diff)

0-29-2
 width 2 != 3 (B2M-000-002-B.diff)
 width 2 != 3 (B3M-000-003-B.diff)
 width 2 != 3 (B3M-000-004-B.diff)
 width 2 != 3 (B3M-000-005-B.diff)

If we believe these mel-bufr tables (and we havent made some mistake in
analysing them), scale/reference/width seems to be allowed to change.
For example, the last field, 0-29-2, bit width was changed from 2 to 3
at version 6. If so, then we have no choice but to maintain versioned
code tables, and assume that the version number (BUFR section 1 octet
11) is correct.

Of course the real question is, what values did the coders use? We have
some hope of automatically detecting when bit width is wrong by counting
bits in the message and comparing to the data size. If scale/offset is
wrong, it will take humans and/or more complex algorithms to detect
that. 
It would be very helpful to hear what the actual practice is with tables
at the various generating centers. Or to hear that the mel-bufr tables
are wrong, or some other flaw in our conclusion.

Thanks for any thoughts,
John



Kellett, Stanley wrote:

John,
First of all I am sorry for late response as I have just gotten back from a camp trip with my family. 

No scale, width and reference values can't be changed once
operational.
What has probably happened is there was a mistake in the last published tables which was corrected. 

Stan

-----Original Message-----
From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx]
Sent: 24 July 2008 18:20
To: Wright, Bruce; Jeff Ator; Ross, Gil; Michelle Mainelli; Scott Jacobs; Kellett, Stanley; Paul Hamer; Chris Macdermaid; Robb Kambic 
Subject: table versions
We are trying to understand how the different versions of the WMO master table differ from each other. We are looking at the "mel-bufr" tables, which seem to be the current best representations of the different versions of the WMO master tables, although we are not sure how they were generated.
We compare these tables against our version of the WMO master table version 13, which we extracted from the Word doc. Typical are the following difference of table 12 with table 13: 


1) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-13-81
3;0;14;S m-1 from B3M-000-012-B
3;0;14;Siemens m-1 from B4M-000-013-B

2) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-15-11
3;14000;13;log from B3M-000-012-B
3;14000;13;log(1/m2) from B4M-000-013-B

3) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-22-39
3;-5000;12;m from B3M-000-012-B
3;-5000;13;m from B4M-000-013-B


Difference 1) and 2) seems to be clarifications of the units. In 1) "S
m-1" has been changed to "Siemens m-1" and in 2) "log" was changed to "log(1/m2)". It seems likely that those changes are clarifications or corrections that should be used in version 12 BUFR messages.
Difference 3) is a change in bit width of field 0-22-39. Should this be understood that version 12 messages use bit width 12, while version 13 uses bit width 13? Obviously this is a critical thing to know. 

When we compare table 10 with 13, we see:

4) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-2-91
0;4;10;a from B3M-000-010-B
4;0;10;a from B4M-000-013-B

5) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-13-60
1;-10;17;kg m-2 from B3M-000-010-B
1;-1;17;kg m-2 from B4M-000-013-B
For difference 4) I might guess that the scale and reference were mistakenly flipped, that is, it corrects a typo in version 10. Possibly difference 5), which changes the reference value from -10 to -1, corrects a typo also?
Are new table versions really allowed to change Scale,Ref,Width or Units of existing table entries? Because if they are, we really have to carefully track what changes with each version.
Are there archives of previous versions of WMO master tables? Are the mel-bufr tables accurate? 

Any thoughts or advice would be appreciated.

_______________________________________________
bufrtables mailing list
bufrtables@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the bufrtables archives: