bufrtables mailing list is no longer active. The list archives are made available for historical reasons.
Hi Jeff: Some comments are embedded. On 3/29/2011 6:50 AM, Jeff Ator wrote:
Hello Everyone, To expand on some previous points... * The WMO does maintain machine-readable versions of the tables, for both BUFR and GRIB, at http://www.wmo.int/pages/prog/www/WMOCodes/TDCFtables.html Thanks to Paolo for pointing this out in his earlier email. Different versions of the tables can be downloaded from this site. For BUFR at least, table version 13 is a superset of all previous versions, so it can be used to decode BUFR messages from all previous versions. This is no longer true beginning with version 14, where it's possible that some deprecated items have now been removed or that some descriptor characteristics have been modified from previous versions of the table. So for BUFR, decoding centers should maintain copies of tables 13, 14 and onward in whatever format(s) are required by their local processing software.
It is important that the WMO maintain canonical copies of versions going forward. Also, one should realize that there may be some errors in these tables, these will get fixed as they are noticed and corrected or clarified. Eventually we will have tables that dont have errors, but then new entries are added and the process starts again. So both encoding and decoding centers will need to keep up with the latest canonical tables. One cant download them once and keep using them for 10 years.
* John rightly points out that the proper use of this table version number by message originators would eliminate the problem outlined in his paper. In my experience, this problem stems mostly from a casual attitude by originators in ensuring they've used the proper version number in their messages. Many originators use software where this number is often hardcoded and so becomes, at best, an afterthought. This is an education issue that WMO is working hard to address among its members.
* There's also a concerted effort among members to develop BUFR templates for certain types of commonly-reported data such as SYNOP, BUOY, TEMP/PILOT, CLIMAT, etc. This is a by-product of WMO's ongoing migration from these old alphanumeric fixed-field formats to BUFR. The list of templates is available athttp://www.wmo.int/pages/prog/www/WMOCodes/TemplateExamples.html#Regulations, and while their use isn't mandatory, it does make things a lot simpler for downstream codes which have to interpret the decoded output, and which is another point that John made in his paper.
this is helpful.one doesnt know for sure that the bufr message is using a template, i think, without actually comparing the message to the template?
* If anyone has a requirement for a new BUFR or GRIB2 descriptor which they feel would be reasonable to propose as a new official WMO descriptor (vs. just using their own local descriptor number), please let me know. I represent the U.S. to the WMO codes group which reviews and approves these types of requests. Depending on the nature of the request, there are fast-track procedures available which can lead to formal approval within a matter of 2-3 months. * As originally envisioned, BUFR and GRIB2 weren't designed to be formats for archive storage, but rather for efficient real-time exchange of meteorological data. Nevertheless, this doesn't mean they can't be used as archive formats. We do this here at NCEP, and the approach we use involves storing a copy of the applicable table with each archived dataset. Note that, for BUFR at least, table information can be encoded into BUFR messages using descriptors from Class 0 of Table B. When this is done, the necessary table information can be easily retained alongside the data in a very compact and efficient manner, using one or two additional BUFR messages at the head of each archived file. Such an approach could even be used when exchanging real-time data sets between centers, at the cost of one or two additional BUFR messages. This would eliminate the problem of receiving centers having to "guess" whether the table version number in each data message was encoded properly.
We arent yet dealing completely with this way of storing the tables in bufr messages, but it certainly solves the problem, as long as those messages are stored in the same file as the messages that they refer to. Assuming this works, I would modify my conclusion that "BUFR/GRIB is not suitable as long-term storage formats" to add "unless the tables are stored in the file, etc".
* In my opinion, when everyone follows the rules (e.g. using official descriptors with proper table version numbers), the process works very well. The trick of course is to get everyone (and their software) to pay attention to the rules. But this is true of any format and is not unique to BUFR and GRIB.
Of course, errors are possible no matter what, but this particular problem is specific to table-driven formats like GRIB and especially BUFR. It doesnt occur, for example, in netCDF when using CF conventions. However, its true that one ultimately refers to human-readable documents for semantics.
With best regards, -Jeff
Thanks for your thoughts! Regards, John