NOTE: The decoders mailing list is no longer active. The list archives are made available for historical reasons.
David, This is discouraging, I spent hours looking through raw bulletins to "try" make the decoder correct. I might try looking at the station ID before processing, don't know if that will help. My philosophy is it's better to disregard bulletins/reports rather then enter "bad" data into a file. That said, your example bulletin should be discarded. ugh. Will let you know about my new ideas. Robb... On Thu, 4 Mar 2004, David Larson wrote:
I do see non-US bulletins that are split "badly" according to this code
change ...
628
SANK31 MNMG 041706
METAR
MNPC 041700Z 08012KT 7000 BKN016 29/26 Q1015
MNRS 041700Z 06010KT 9999 FEW022 BKN250 30/23 Q1012
MNJG 041700Z 36004KT 7000 VCRA BKN016 22/17 Q1015
MNJU 041700Z 10006KT 9999 SCT025 32/19 Q1013
MNCH 041700Z 02010KT 9999 SCT030 33/21 Q1012
MNMG 041700Z 07016KT 9999 SCT025 32/20 Q1011 A2988
MNBL 041700Z 10008KT 9999 SCT019 SCT070 29/25 Q1014
Not only is each line a new report, but what worsens it is that the
*last* entry *is* separated by an equal! Yuck. Perhaps this is just a
junk bulletin? I'm suprised that it could even go out this way.
Does anyone you know even make an attempt to use the perl metar decoder
for non-US stations? I've tried long enough to estimate the work as a
*lot*.
Dave
David Larson wrote:
> I've looked into this problem, which I didn't know existed.
>
> Your code is now:
>
> # Separate bulletins into reports
> if( /=\n/ ) {
> s#=\s+\n#=\n#g ;
> @reports = split( /=\n/ ) ;
> } else {
> #@reports = split ( /\n/ ) ;
> s#\n# #g ;
> next if( /\d{4,6}Z.*\d{4,6}Z/ ) ;
> $reports[ 0 ] = $_ ;
> }
>
> But based on your assumption that these bulletins will not, and
> cannot, contain multiple reports (which seems and appears to be
> reasonable), then there really only needs to be one split, right?
> Because if there is no equal, the entire line will be placed into the
> first report. This seems to be a slight simplification:
>
> # Separate bulletins into reports
> if( /=\n/ ) {
> s#=\s+\n#=\n#g ;
> } else {
> s#\n# #g ;
> }
> @reports = split( /=\n/ ) ;
> ... snip ... the next line is placed down many lines
> next if( /\d{4,6}Z.*\d{4,6}Z/ ) ;
>
> Also, it is an error to have multiple time specifications in any
> report, right? So that can be generalized as well, as I have done above.
>
> You asked for my comments, and well, there you have them! :-) I
> might take a closer look at the rest of the changes as well, but that
> will be delayed a bit.
>
> I sure appreciate your quick responses to all correspondence.
>
> Dave
>
> Robb Kambic wrote:
>
>> David,
>>
>> Yes, I know about the problem. The problem exists in bulletins that
>> don't
>> use the = sign to seperate reports. The solution is to assume that
>> bulletins
>> that don't use = only have one report. I scanned many raw reports and
>> this
>> seems to be true, so I changed the code to:
>>
>> < @reports = split ( /\n/ ) ;
>> ---
>>
>>
>>> #@reports = split ( /\n/ ) ;
>>> s#\n# #g ;
>>> next if( /\d{4,6}Z.*\d{4,6}Z/ ) ;
>>> $reports[ 0 ] = $_;
>>>
>>
>>
>> The new code is attached. I'm also working on a newer version of the
>> decoder, it's in the ftp decoders directory. ie
>>
>> metar2nc.new and metar.cdl.new
>>
>> The pqact.conf entry needs to change \2:yy to \2:yyyy because it now
>> uses
>> the century too. The cdl is different, merges vars that have different
>> units into one. ie wind knots, mph, and m/s are all store using winds
>> m/s. Also, store all reports per station into one record. Take a
>> look, I
>> would appreciate any comments before it's released.
>>
>> Robb...
>>
>>
>> On Tue, 2 Mar 2004, David Larson wrote:
>>
>>
>>
>>> Robb,
>>>
>>> I've been chasing down a problem that seems to cause perfectly good
>>> reports to be discarded by the perl metar decoder. There is a comment
>>> in the 2.4.4 decoder that reads "reports appended together wrongly",
>>> the
>>> code in this area takes the first line as the report to process, and
>>> discards the next line.
>>>
>>> To walk through this, I'll refer to the following report:
>>>
>>> 132
>>> SAUS80 KWBC 021800 RRD
>>> METAR
>>> K4BL 021745Z 12005KT 3SM BR OVC008 01/M01 RMK SLP143 NOSPECI 60011
>>> 8/2// T00061006 10011 21017 51007
>>>
>>> The decoder attempts to classify the report type ($rep_type on line 257
>>> of metar2nc), in doing so, it classifies this report as a "SPECI" ...
>>> which isn't what you'd expect by visual inspection of the report.
>>> However, perl is doing the right thing given that it is asked to match
>>> on #(METAR|SPECI) \d{4,6}Z?\n# which exists in the remarks of the
>>> report.
>>>
>>> The solution is probably to bind the text to the start of the line with
>>> a caret. Seems to work pretty well so far.
>>>
>>> I've changed the lines (257-263) in metar2nc-v2.4.4 from:
>>>
>>> if( s#(METAR|SPECI) \d{4,6}Z?\n## ) {
>>> $rep_type = $1 ;
>>> } elsif( s#(METAR|SPECI)\s*\n## ) {
>>> $rep_type = $1 ;
>>> } else {
>>> $rep_type = "METAR" ;
>>> }
>>>
>>> To:
>>>
>>> if( s#^(METAR|SPECI) \d{4,6}Z?\n## ) {
>>> $rep_type = $1 ;
>>> } elsif( s#^(METAR|SPECI)\s*\n## ) {
>>> $rep_type = $1 ;
>>> } else {
>>> $rep_type = "METAR" ;
>>> }
>>>
>>> I simply added the caret (^) to bind the pattern to the start of the
>>> report.
>>>
>>> Let me know what you think.
>>> Dave
>>>
>>>
>>
>> ------------------------------------------------------------------------
>>
>> #! /usr/local/bin/perl
>> #
>> # usage: metar2nc cdlfile [datatdir] [yymm] < ncfile
>> #
>> #
>> #chdir( "/home/rkambic/code/decoders/src/metar" ) ;
>>
>> use NetCDF ;
>> use Time::Local ;
>> # process command line switches
>> while ($_ = $ARGV[0], /^-/) {
>> shift;
>> last if /^--$/;
>> /^(-v)/ && $verbose++;
>> }
>> # process input parameters
>> if( $#ARGV == 0 ) {
>> $cdlfile = $ARGV[ 0 ] ;
>> } elsif( $#ARGV == 1 ) {
>> $cdlfile = $ARGV[ 0 ] ;
>> if( $ARGV[ 1 ] =~ /^\d/ ) {
>> $yymm = $ARGV[ 1 ] ;
>> } else {
>> $datadir = $ARGV[ 1 ] ;
>> }
>> } elsif( $#ARGV == 2 ) {
>> $cdlfile = $ARGV[ 0 ] ;
>> $datadir = $ARGV[ 1 ] ;
>> $yymm = $ARGV[ 2 ] ;
>> } else {
>> die "usage: metar2nc cdlfile [datatdir] [yymm] < ncfile $!\n" ;
>> }
>> print "Missing cdlfile file $cdlfile: $!\n" unless -e $cdlfile ;
>>
>> if( -e "util/ncgen" ) {
>> $ncgen = "util/ncgen" ;
>> } elsif( -e "/usr/local/ldm/util/ncgen" ) {
>> $ncgen = "/usr/local/ldm/util/ncgen" ;
>> } elsif( -e "/upc/netcdf/bin/ncgen" ) {
>> $ncgen = "/upc/netcdf/bin/ncgen" ;
>> } elsif( -e "./ncgen" ) {
>> $ncgen = "./ncgen" ;
>> } else {
>> open( NCGEN, "which ncgen |" ) ;
>> $ncgen = <NCGEN> ;
>> close( NCGEN ) ;
>>
>> if( $ncgen =~ /no ncgen/ ) {
>> die "Can't find NetCDF utility 'ncgen' in PATH, util/ncgen
>> /usr/local/ldm/util/ncgen, /upc/netcdf/bin/ncgen, or ./ncgen : $!\n" ;
>> } else {
>> $ncgen = "ncgen" ;
>> }
>> }
>> # the data and the metadata directories $datadir = "." if( ! $datadir
>> ) ;
>> $metadir = $datadir . "/../metadata/surface/metar" ;
>> # redirect STDOUT and STDERR
>> open( STDOUT, ">$datadir/metarLog.$$.log" ) ||
>> die "could not open $datadir/metarLog.$$.log: $!\n" ;
>> open( STDERR, ">&STDOUT" ) ||
>> die "could not dup stdout: $!\n" ;
>> select( STDERR ) ; $| = 1 ;
>> select( STDOUT ) ; $| = 1 ;
>>
>> die "Missing cdlfile file $cdlfile: $!\n" unless -e $cdlfile ;
>>
>> # year and month
>> if( ! $yymm ) {
>> $theyear = (gmtime())[ 5 ] ;
>> $theyear = ( $theyear < 100 ? $theyear : $theyear - 100 ) ;
>> $theyear = sprintf( "%02d", $theyear ) ;
>> $themonth = (gmtime())[ 4 ] ;
>> $themonth++ ;
>> $yymm = $theyear . sprintf( "%02d", $themonth ) ;
>> } else {
>> $theyear = substr( $yymm, 0, 2 ) ;
>> $themonth = substr( $yymm, 2 ) ;
>> }
>> # file used for bad metars or prevention of overwrites to ncfiles
>> open( OPN, ">>$datadir/rawmetars.$$.nc" ) || die "could not
>> open $datadir/rawmetars.$$.nc: $!\n" ;
>> # set error handling to verbose only
>> $result = NetCDF::opts( VERBOSE ) ;
>>
>> # set interrupt handler
>> $SIG{ 'INT' } = 'atexit' ;
>> $SIG{ 'KILL' } = 'atexit' ;
>> $SIG{ 'TERM' } = 'atexit' ;
>> $SIG{ 'QUIT' } = 'atexit' ;
>>
>> # set defaults
>>
>> $F = -99999 ;
>> $A = \$F ;
>> $S1 = "\0" ;
>> $AS1 = \$S1 ;
>> $S2 = "\0\0" ;
>> $AS2 = \$S2 ;
>> $S3 = "\0\0\0" ;
>> $AS3 = \$S3 ;
>> $S4 = "\0\0\0\0" ;
>> $AS4 = \$S4 ;
>> $S8 = "\0" x 8 ;
>> $AS8 = \$S8 ;
>> $S10 = "\0" x 10 ;
>> $AS10 = \$S10 ;
>> $S15 = "\0" x 15 ;
>> $AS15 = \$S15 ;
>> $S32 = "\0" x 32 ;
>> $AS32 = \$S32 ;
>> $S128 = "\0" x 128 ;
>> $AS128 = \$S128 ;
>>
>> %CDL = (
>> "rep_type", 0, "stn_name", 1, "wmo_id", 2, "lat", 3, "lon", 4,
>> "elev", 5,
>> "ob_hour", 6, "ob_min", 7, "ob_day", 8, "time_obs", 9,
>> "time_nominal", 10, "AUTO", 11, "UNITS", 12, "DIR", 13, "SPD", 14,
>> "GUST", 15, "VRB", 16, "DIRmin", 17, "DIRmax", 18, "prevail_VIS_SM",
>> 19, "prevail_VIS_KM", 20, "plus_VIS_SM", 21, "plus_VIS_KM", 22,
>> "prevail_VIS_M", 23, "VIS_dir", 24, "CAVOK", 25, "RVRNO", 26,
>> "RV_designator", 27, "RV_above_max", 28, "RV_below_min", 29,
>> "RV_vrbl", 30, "RV_min", 31, "RV_max", 32, "RV_visRange", 33, "WX",
>> 34, "vert_VIS", 35, "cloud_type", 36, "cloud_hgt", 37,
>> "cloud_meters", 38, "cloud_phenom", 39, "T", 40, "TD", 41,
>> "hectoPasc_ALTIM", 42, "inches_ALTIM", 43, "NOSIG", 44,
>> "TornadicType", 45, "TornadicLOC", 46, "TornadicDIR", 47,
>> "BTornadic_hh", 48, "BTornadic_mm", 49,
>> "ETornadic_hh", 50, "ETornadic_mm", 51, "AUTOindicator", 52,
>> "PKWND_dir", 53, "PKWND_spd", 54, "PKWND_hh", 55, "PKWND_mm", 56,
>> "WshfTime_hh", 57, "WshfTime_mm", 58, "Wshft_FROPA", 59, "VIS_TWR", 60,
>> "VIS_SFC", 61, "VISmin", 62, "VISmax", 63, "VIS_2ndSite", 64,
>> "VIS_2ndSite_LOC", 65, "LTG_OCNL", 66, "LTG_FRQ", 67, "LTG_CNS", 68,
>> "LTG_CG", 69, "LTG_IC", 70, "LTG_CC", 71, "LTG_CA", 72, "LTG_DSNT", 73,
>> "LTG_AP", 74, "LTG_VcyStn", 75, "LTG_DIR", 76, "Recent_WX", 77,
>> "Recent_WX_Bhh", 78, "Recent_WX_Bmm", 79, "Recent_WX_Ehh", 80,
>> "Recent_WX_Emm", 81, "Ceiling_min", 82, "Ceiling_max", 83,
>> "CIG_2ndSite_meters", 84, "CIG_2ndSite_LOC", 85, "PRESFR", 86,
>> "PRESRR", 87,
>> "SLPNO", 88, "SLP", 89, "SectorVIS_DIR", 90, "SectorVIS", 91, "GR", 92,
>> "GRsize", 93, "VIRGA", 94, "VIRGAdir", 95, "SfcObscuration", 96,
>> "OctsSkyObscured", 97, "CIGNO", 98, "Ceiling_est", 99, "Ceiling", 100,
>> "VrbSkyBelow", 101, "VrbSkyLayerHgt", 102, "VrbSkyAbove", 103,
>> "Sign_cloud", 104, "Sign_dist", 105, "Sign_dir", 106, "ObscurAloft",
>> 107,
>> "ObscurAloftSkyCond", 108, "ObscurAloftHgt", 109, "ACFTMSHP", 110,
>> "NOSPECI", 111, "FIRST", 112, "LAST", 113, "Cloud_low", 114,
>> "Cloud_medium", 115, "Cloud_high", 116, "SNINCR", 117,
>> "SNINCR_TotalDepth", 118,
>> "SN_depth", 119, "SN_waterequiv", 120, "SunSensorOut", 121,
>> "SunShineDur", 122,
>> "PRECIP_hourly", 123, "PRECIP_amt", 124, "PRECIP_24_amt", 125,
>> "T_tenths", 126,
>> "TD_tenths", 127, "Tmax", 128, "Tmin", 129, "Tmax24", 130, "Tmin24",
>> 131, "char_Ptend", 132, "Ptend", 133, "PWINO", 134, "FZRANO", 135,
>> "TSNO", 136, "PNO", 137, "maintIndicator", 138, "PlainText", 139,
>> "report", 140, "remarks", 141 ) ;
>>
>> # default netCDF record structure, contains all vars for the METAR
>> reports
>> @defaultrec = ( $A, $A, $A, $A, $A, $A, $A, $A, $A, $A, $A, $A, $AS3,
>> $A, $A,
>> $A, $A, $A, $A, $A, $A, $A, $A, $A, $AS2, $A, $A, [( $S3, $S3, $S3,
>> $S3 )],
>> [( $F, $F, $F, $F )], [( $F, $F, $F, $F )], [( $F, $F, $F, $F )], [(
>> $F, $F, $F, $F )], [( $F, $F, $F, $F )], [( $F, $F, $F, $F )], $AS32,
>> $A,
>> [( $S4, $S4, $S4, $S4, $S4, $S4 )], [( $F, $F, $F, $F, $F, $F )],
>> [( $F, $F, $F, $F, $F, $F )], [( $S4, $S4, $S4, $S4, $S4, $S4 )],
>> $A, $A, $A, $A, $A, $AS15, $AS10, $AS2, $A, $A, $A, $A, $AS4, $A, $A,
>> $A, $A, $A, $A, $A, $A, $A, $A, $A, $A, $AS10, $A, $A, $A, $A, $A,
>> $A, $A, $A, $A, $A, $AS2, [( $S8, $S8, $S8 )], [( $F, $F, $F )],
>> [( $F, $F, $F )], [( $F, $F, $F )], [( $F, $F, $F )], $A, $A, $A, $A,
>> $A, $A, $A, $A, $AS2, $A, $A, $A, $A, $AS2, $AS8, $A, $A, $A, $A,
>> $AS3, $A, $AS3, $AS10, $AS10, $AS10, $AS8, $AS3, $A, $A, $A, $A, $A,
>> $AS1, $AS1, $AS1, $A, $A, $A, $A, $A, $A, $A, $A, $A, $A,
>> $A, $A, $A, $A, $A, $A, $A, $A, $A, $A, $A, $A, $AS128, $AS128,
>> $AS128 ) ;
>>
>> # two fold purpose array, if entry > 0, then var is requested and
>> it's value
>> # is the position in the record, except first entry
>> @W = ( 0 ) x ( $#defaultrec +1 ) ;
>> $W[ 0 ] = -1 ;
>>
>> # open cdl and create record structure according to variables
>> open( CDL, "$cdlfile" ) || die "could not open $cdlfile: $!\n" ;
>> $i = 0 ;
>> while( <CDL> ) {
>> if( s#^\s*(char|int|long|double|float) (\w{1,25})## ) {
>> ( $number ) = $CDL{ $2 } ;
>> push( @rec, $defaultrec[ $number ] ) ;
>> $W[ $number ] = $i++ ;
>> }
>> }
>> close CDL ;
>> undef( @defaultrec ) ;
>> undef( %CDL ) ;
>>
>> # read in station data
>> if( -e "etc/sfmetar_sa.tbl" ) {
>> $sfile = "etc/sfmetar_sa.tbl" ;
>> } elsif( -e "./sfmetar_sa.tbl" ) {
>> $sfile = "./sfmetar_sa.tbl" ;
>> } else {
>> die "Can't find sfmetar_sa.tbl station file.: $!\n" ;
>> }
>> open( STATION, "$sfile" ) || die "could not open $sfile: $!\n" ;
>>
>> while( <STATION> ) {
>> s#^(\w{3,6})?\s+(\d{4,5}).{40}## ;
>> $id = $1 ;
>> $wmo_id = $2 ;
>> $wmo_id = "0" . $wmo_id if( length( $wmo_id ) == 4 ) ;
>> ( $lat, $lon, $elev ) = split ;
>> $lat = sprintf( "%7.2f", $lat / 100 ) ;
>> $lon = sprintf( "%7.2f", $lon / 100) ;
>>
>> # set these vars ( $wmo_id, $lat, $lon, $elev ) $STATIONS{
>> "$id" } = "$wmo_id $lat $lon $elev" ;
>> }
>> close STATION ;
>>
>> # read in list of already processed reports if it exists
>> # open metar.lst, list of reports processed in the last 4 hours.
>> if( -e "$datadir/metar.lst" ) {
>> open( LST, "$datadir/metar.lst" ) || die "could not open
>> $datadir/metar.lst: $!\n" ;
>> while( <LST> ) {
>> ( $stn, $rtptime, $hr ) = split ;
>> $reportslist{ "$stn $rtptime" } = $hr ;
>> }
>> close LST ;
>> #unlink( "$datadir/metar.lst" ) ;
>> }
>> # Now begin parsing file and decoding observations breaking on cntrl C
>> $/ = "\cC" ;
>>
>> # set select processing here from STDIN
>> START:
>> while( 1 ) {
>> open( STDIN, '-' ) ;
>> vec($rin,fileno(STDIN),1) = 1;
>> $timeout = 1200 ; # 20 minutes
>> $nfound = select( $rout = $rin, undef, undef, $timeout );
>> # timed out
>> if( ! $nfound ) {
>> print "Shut down, time out 20 minutes\n" ;
>> &atexit() ;
>> }
>> &atexit( "eof" ) if( eof( STDIN ) ) ;
>>
>> # Process each line of metar bulletins, header first
>> $_ = <STDIN> ;
>> #next unless /METAR|SPECI/ ;
>> s#\cC## ;
>> s#\cM##g ;
>> s#\cA\n## ;
>> s#\c^##g ;
>>
>> s#\d\d\d \n## ;
>> s#\w{4}\d{1,2} \w{4} (\d{2})(\d{2})(\d{2})?.*\n## ;
>> $tday = $1 ;
>> $thour = $2 ;
>> $thour = "23" if( $thour eq "24" ) ;
>> $tmin = $3 ;
>> $tmin = "00" unless( $tmin ) ;
>> next unless ( $tday && defined( $thour ) ) ;
>> $time_trans = thetime( "trans" ) ;
>> if( s#(METAR|SPECI) \d{4,6}Z?\n## ) {
>> $rep_type = $1 ;
>> } elsif( s#(METAR|SPECI)\s*\n## ) {
>> $rep_type = $1 ;
>> } else {
>> $rep_type = "METAR" ;
>> }
>> # Separate bulletins into reports if( /=\n/ ) {
>> s#=\s+\n#=\n#g ;
>> @reports = split( /=\n/ ) ;
>> } else {
>> #@reports = split ( /\n/ ) ;
>> s#\n# #g ;
>> next if( /\d{4,6}Z.*\d{4,6}Z/ ) ;
>> $reports[ 0 ] = $_ ;
>> }
>>
>>
>
============================================================================== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research rkambic@xxxxxxxxxxxxxxxx WWW: http://www.unidata.ucar.edu/ ==============================================================================
decoders archives: