[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Was ldmConnect deprecated?



Hi Robb,

Was it deprecates because the NWSTG/TOC/Gateway doesn't send out segmented products anymore, or does other software piece together the segmented products?

Thanks,
Gregg

Robb Kambic wrote:
On Wed, 6 Oct 2010, Gregory Grosshans wrote:

Can you tell me if ldmConnect was deprecated?

Hi Gregory,

Yes it was deprecated. But, I attached the documentation and decoder ldmConnect. It should work but it's not official supported.

Robb...

I wasn't able to easily find
information about this, except it was in the decoders-4.1.7 release and then
was dropped in the 5.0 release.

Thanks,
Gregg



===============================================================================
Robb Kambic                       Unidata Program Center
Software Engineer III               Univ. Corp for Atmospheric Research
address@hidden           WWW: http://www.unidata.ucar.edu/
===============================================================================

#!/usr/local/bin/perl ## Connect split LDM bulletins #use strict ; package Part ; use File::Path ; sub new { return bless {} ; } sub getData { my $self = shift; my( $data ) = @_; my( $count, $state, $wmohdr ) ; return "SKIP", "", "" if( length( $data ) < 5 ) ; if( ( $count = $data =~ tr/\n/\n/ ) < 7 ) { # not enough CR's $data =~ s#\cM+\n?#\cM\cM\n#g ; # convert \cM+ to CR's } $data =~ s#\cC## ; $data =~ s#\cA\cM+\n## ; # get header $data =~ s#(\d\d\d)\s*\cM+\n## ; $self->{ 'number' } = $1 ; $data =~ s#^\s*\cM+\n## ; # reports with extra CR return "SKIP", "", "" if( length( $data ) < 5 ) ; $state = "" ; $data =~ m#(.*?\cM+)\n# ; $wmohdr = $1 ; if( $wmohdr =~ m# (P\w\w)\cM| (P\w\w) (RTD|RTN)# ) { $state = ( $1 ? $1 : $2 ) ; } if( $data =~ s#^(\w{4} ?\d{0,2})\s+(\w{4})\s+(\d{4})(\d{2}).*\cM+\n##m ) { $self->{ 'bulletin' } = $1 ; $self->{ 'bulletin' } =~ s#\s+## ; $self->{ 'station' } = $2 ; $self->{ 'header' } = "$1 $2 $3$4" ; $self->{ 'ddhh' } = $3 ; $self->{ 'mm' } = $4 ; } elsif( $data =~ s#(\w{4} ?\d{0,2})\s+(\w{4})\s+(DDHHMM).*\cM+\n##m ) { $self->{ 'bulletin' } = $1 ; $self->{ 'bulletin' } =~ s#\s+## ; $self->{ 'station' } = $2 ; $self->{ 'header' } = "$1 $2 $3" ; $self->{ 'ddhh' } = "DDHH" ; $self->{ 'mm' } = "MM" ; } else { $self->{ 'ddhh' } = "DDHH" ; $self->{ 'mm' } = "MM" ; } if( ! $state ) { $self->{ 'data' } = $data ; return "OK", "$self->{ 'header' }", "$state" ; } elsif( $state =~ /^PZ/ ) { $self->{ "$state" } = $data ; return "DONE", "$self->{ 'header' }", "$state" ; } elsif( $state =~ /^PAA/ ) { $self->{ 'parts' } = "$state " ; $self->{ 'btype' } = $1 if( $data =~ s#^(\w{5,7})\s*\cM*\n## ) ; $self->{ "$state" } = $data ; return "HEAD", "$self->{ 'header' }", "$state" ; } elsif( $state =~ /^P/ ) { $self->{ "$state" } = $data ; return "PART", "$self->{ 'header' }", "$state" ; } else { #if( $state =~ /AMD|COR|AAA|RTD/ ) $self->{ 'data' } = $data ; return "OK", "$self->{ 'header' }", "$state" ; } } # end getData # build filename sub buildFN { my $self = shift; my( $dirStruct, $datadir, $yymm ) = @_ ; my( $filename ) ; if( $dirStruct ) { $filename = $datadir . "/" . $self->{ 'station' } . "/" . $self->{ 'bulletin' } ; mkpath( $filename, 0, 0775 ) ; $filename .= "/" . $yymm . $self->{ 'ddhh' } . $self->{ 'mm' } ; } else { $filename = $datadir . "/" . $yymm . $self->{ 'ddhh' } ; } return $filename ; } # End buildFN sub print { my $self = shift; my( $cChars, $section ) = @_ ; my( $end ) ; if( $cChars ) { $end = "\n" ; $self->{ 'data' } =~ s#\cM##g ; } else { $end = "\cM\cM\n" ; print "\cA\cM\cM\n" ; } print $self->{ 'number' }, " $end" ; if( defined( $self->{ 'data' } ) ) { print "$self->{ 'header' }$end" ; print "$self->{ 'data' }$end" ; } else { print "$self->{ 'header' } $section$end" ; print $self->{ $section } . $end ; } print "\cC" unless( $cChars ) ; } package Bulletin ; #use vars qw( @ISA ) ; @ISA = qw( Part ) ; sub new { my $type = shift ; my( $self ) = @_ ; $self->{ 'Time' } = time() ; # Time of existence bless( $self, $type ) ; return $self ; } # end new sub time { # returns number of seconds in existence my $self = shift; return time() - $self->{ 'Time' } ; } # add parts into bulletin object sub addPart { my $self = shift; my( $section, $part ) = @_ ; if( ! defined( $self->{ 'number' } ) ) { $self->{ 'number' } = $part->{ 'number' } ; $self->{ 'header' } = $part->{ 'header' } ; $self->{ 'ddhh' } = $part->{ 'ddhh' } ; $self->{ 'btype' } = $part->{ 'btype' } ; } $self->{ 'parts' } .= "$section " unless( $self->{ 'parts' } =~ /$section/ ) ; $self->{ "$section" } = $part->{ "$section" } ; } # end addPart # PAA section is already collected sub PAA { my $self = shift; if( $self->{ 'parts' } =~ /PAA/ ) { return 1 ; } else { return 0 ; } } # end PAA # combine sections together to make complete bulletin sub combine { my $self = shift; my( $section, @sections ) ; @sections = split( /[ \t\n]+/, $self->{ 'parts' } ) ; foreach $section ( sort @sections ) { if( $self->{ 'btype' } && $section =~ /PAA/ ) { $self->{ "$section" } = "$self->{ 'btype' } \cM\n" . $self->{ "$section" } ; } elsif( $self->{ 'btype' } ) { $self->{ "$section" } =~ s#^$self->{ 'btype' }\s*\cM+\n## ; } # eat ending line feed in case bulletin split in middle of word $self->{ "$section" } =~ s/\cM+\n$//s ; $self->{ 'data' } .= $self->{ "$section" } ; } } # end combine sub print { my $self = shift; my( $cChars ) = @_ ; my( $end ) ; if( $cChars ) { $end = "\n" ; $self->{ 'data' } =~ s#\cM##g ; } else { $end = "\cM\cM\n" ; print "\cA\cM\cM\n" ; } print $self->{ 'number' }, " $end" ; print "$self->{ 'header' }$end" ; print "$self->{ 'data' }$end" ; print "\cC" unless( $cChars ) ; } # fill bulletins parts from inProgress file sub fill { my $self = shift; my( $data ) = @_ ; my( $i, @parts, $section, @sections ) ; $i = 0 ; ( @sections ) = split( /\t/, $data ) ; $self->{ 'number' } = $sections[ $i++ ] ; $self->{ 'bulletin' } = $sections[ $i++ ] ; $self->{ 'station' } = $sections[ $i++ ] ; $self->{ 'header' } = $sections[ $i++ ] ; $self->{ 'ddhh' } = $sections[ $i++ ] ; $self->{ 'mm' } = $sections[ $i++ ] ; $self->{ 'parts' } = $sections[ $i++ ] ; ( @parts ) = split( /[ \t\n]+/, $self->{ 'parts' } ) ; foreach $section ( @parts ) { $self->{ "$section" } = $sections[ $i++ ] ; } return $self->{ 'header' } ; } # dump bulletins parts to inProgress file sub dump { my $self = shift; my( $section, @sections ) ; print $self->{ 'number' }, "\t" ; print $self->{ 'bulletin' }, "\t" ; print $self->{ 'station' }, "\t" ; print "$self->{ 'header' }\t" ; print "$self->{ 'ddhh' }\t" ; print "$self->{ 'mm' }\t" ; print "$self->{ 'parts' }\t" ; @sections = split( /[ \t\n]+/, $self->{ 'parts' } ) ; foreach $section ( sort @sections ) { print $self->{ "$section" }, "\t" ; } print "\cC" ; } # sets, deletes, and stores file handles package doIO ; use FileHandle ; #use vars qw( %handles %hTimes $logging ) ; %handles = () ; %hTimes = () ; $logging = 0 ; sub new { return bless {} ; } sub setLog { my $self = shift; my( $datadir ) = @_ ; # redirect STDOUT and STDERR open( STDOUT, ">$datadir/connectLog.$$.log" ) || die "could not open $datadir/connectLog.$$.log: $!\n" ; open( STDERR, ">&STDOUT" ) || die "could not dup stdout: $!\n" ; select( STDERR ) ; $| = 1 ; select( STDOUT ) ; $| = 1 ; $logging = 1 ; } # sets a filehandle for the data sub setFH { my $self = shift; my( $filename, $ext ) = @_ ; my( $fh, $file, @files, $outfile, $opentime, $thetime, $Time ) ; if( defined( $handles{ "$filename" } ) ) { $fh = $handles{ "$filename" } ; select( $fh ) ; $| = 1 ; return ; } $outfile = $filename . $ext ; # current time $thetime = time() ; ( @files ) = keys %handles ; #print STDERR "Number of files open is $#files.\n" ; if( $#files < 15 ) { $opentime = 1200 ; } elsif( $#files < 25 ) { $opentime = 600 ; } elsif( $#files < 35 ) { $opentime = 300 ; } elsif( $#files < 45 ) { $opentime = 60 ; } else { $opentime = -1 ; } # close files with no activity for $opentime minutes foreach $file ( keys %handles ) { $Time = $hTimes{ "$file" } ; if( $thetime - $Time > $opentime ) { print STDOUT "Closing $file$ext, No write for > $opentime Sec\n" if( $logging ) ; $fh = $handles{ "$file" } ; close( $fh ) ; delete( $handles{ $file } ) ; } else { $hTimes{ "$file" } = $thetime ; } } # open or create outfiles $fh = new FileHandle ; $fh->open( ">>$outfile" ) || print STDERR "Can't open $outfile: $!\n" ; select( $fh ) ; $fh->autoflush( 1 ) ; $handles{ "$filename" } = $fh ; $hTimes{ "$filename" } = $thetime ; print STDOUT "Opening $outfile at $thetime\n" if( $logging ) ; return ; } # end setFH sub Close { my $self = shift; my( $ext ) = @_ ; my( $fh, $file ) ; foreach $file ( keys %handles ) { $fh = $handles{ "$file" } ; close( $fh ) ; print STDOUT "Closing $file$ext\n" if( $logging ) ; } close( STDOUT ) ; close( STDERR ) ; } sub inProgress { my $self = shift; my( $filename, $task ) = @_ ; if( $task eq "data" ) { $_ = <INPROGRESS> ; } elsif( $task eq "write" ) { open( INPROGRESS, ">$filename" ) ; select( INPROGRESS ) ; $| = 1 ; } elsif( $task eq "read" ) { if( -e "$filename" && -s "$filename" ) { open( INPROGRESS, "$filename" ) ; return 1 ; } else { return 0 ; } } else { close( INPROGRESS ) ; } } #end inProgress # Connect and write bulletins here. package main ; use File::Path ; #use vars qw( $cChars $ext $dirStruct $logging $yymm $datadir $filename ) ; #use vars qw( $status $header $section $io $part $bulletin %inProgress ) ; #use vars qw( $rin $timeout $nfound $rout ) ; # process command line switches while ($_ = $ARGV[0], /^-/) { shift; last if /^--$/; /^(-s)/ && $cChars++ ; # strip control chars # use this extension, default _for.wmo /^(-e)/ && do { # LDM doesn't pass a null, could get next arg $ext = shift ; if( $ext =~ m#^(-|\d|\s+)|/# ) { unshift( @ARGV, $ext ) unless( $ext =~ m#^\s# ) ; $ext = "" ; } } ; /^(-a)/ && $dirStruct++ ; # write /stn/bulletin/report /^(-l)/ && $logging++ ; # turn logging on } $cChars = 0 if( ! defined( $cChars ) ) ; $dirStruct = 0 if( ! defined( $dirStruct ) ) ; $ext = "_for.wmo" if( ! defined( $ext ) ) ; # process input parameters if( $#ARGV == 0 ) { if( $ARGV[ 0 ] =~ /^[0-9]/ ) { $yymm = $ARGV[ 0 ] ; } else { $datadir = $ARGV[ 0 ] ; } } elsif( $#ARGV == 1 ) { $yymm = $ARGV[ 0 ] ; $datadir = $ARGV[ 1 ] ; } else { die "Usage with input on STDIN: ldmConnect [-a] [-s] [-l] [-e ext ] [yymm] [datadir]\n"; } # set interrupt handler $SIG{ 'INT' } = 'atexit' ; $SIG{ 'KILL' } = 'atexit' ; $SIG{ 'TERM' } = 'atexit' ; $SIG{ 'QUIT' } = 'atexit' ; $SIG{ 'HUP' } = 'atexit' ; # the directory $datadir = "." if( ! $datadir ) ; mkpath( $datadir, 0, 0775 ) if( ! -d $datadir ) ; $io = new doIO() ; $io->setLog( $datadir ) if( $logging ) ; # year and month if( ! $yymm ) { my( $theyear, $themonth ) ; $theyear = (gmtime())[ 5 ] ; $theyear = ( $theyear < 100 ? $theyear : $theyear - 100 ) ; $theyear = sprintf( "%02d", $theyear ) ; $themonth = (gmtime())[ 4 ] ; $themonth++ ; $yymm = $theyear . sprintf( "%02d", $themonth ) ; } # Now begin parsing file and decoding observations breaking on cntrl C $/ = "\cC" ; # read in inprogress data from last process, fill in bulletin $filename = $datadir . "/inProgress" . "$ext" ; if( $io->inProgress( $filename, "read" ) ) { while( 1 ) { $io->inProgress( $filename, "data" ) ; last if( !$_ ) ; $part = new Part() ; $bulletin = new Bulletin( $part ) ; $header = $bulletin->fill( $_ ) ; $inProgress{ "$header" } = $bulletin ; undef( $part ) ; } $io->inProgress( $filename, "close" ) ; } # main loop set select processing here from STDIN START: while( 1 ) { open( STDIN, '-' ) ; vec( $rin, fileno( STDIN ), 1 ) = 1 ; $timeout = 1200 ; # 20 minutes $nfound = select( $rout = $rin, undef, undef, $timeout ) ; # timed out if( ! $nfound ) { print STDOUT "Shut down, time out 20 minutes\n" if( $logging ) ; atexit() ; } atexit( "eof" ) if( eof( STDIN ) ) ; $part = new Part() ; # Process data first as part of a bulletin $_ = <STDIN> ; ( $status, $header, $section ) = $part->getData( $_ ) ; if( $status eq "OK" ) { # complete bulletin, write it out $filename = $part->buildFN( $dirStruct, $datadir, $yymm ) ; $io->setFH( $filename, $ext ) ; $part->print( $cChars, $section ) ; } elsif( $status eq "HEAD" ) { # create new bulletin $bulletin = new Bulletin( $part ) ; $inProgress{ "$header" } = $bulletin ; } elsif( $status eq "PART" || $status eq "DONE" ) { if( defined( $inProgress{ "$header" } ) ) { $bulletin = $inProgress{ "$header" } ; $bulletin->addPart( $section, $part ) ; # combine parts and write if( $status eq "DONE" || $bulletin->time() > 900 ) { $bulletin->combine() ; $filename = $bulletin->buildFN ( $dirStruct, $datadir, $yymm ) ; $io->setFH( $filename, $ext ) ; $bulletin->print( $cChars ) ; delete( $inProgress{ "$header" } ) ; } } else { # catches weird PRE, PMD, etc suffices $filename = $part->buildFN ( $dirStruct, $datadir, $yymm ) ; $io->setFH( $filename, $ext ) ; $part->print( $cChars, $section ) ; } } undef( $part ) ; atexit( "eof" ) if( eof( STDIN ) ) ; } # end while( 1 ) atexit( "eof" ); exit( 0 ) ; # should never get here # execute at exit sub atexit { my( $sig ) = @_ ; if( $sig eq "eof" ) { print STDOUT "eof on STDIN --shutting down\n" if( $logging ) ; } elsif( defined( $sig )) { print STDOUT "Caught SIG$sig --shutting down\n" if( $logging ) ; } # check if bulletin is older than 15 minutes, if so write, else save in # the inProgress file $filename = $datadir . "/inProgress" . "$ext" ; $io->inProgress( $filename, "write" ) ; foreach $header ( %inProgress ) { $bulletin = $inProgress{ "$header" } ; if( defined( $bulletin ) ) { if( $bulletin->time() < 900 ) { $bulletin->dump() ; } else { $bulletin->combine() ; $filename = $bulletin->buildFN ( $dirStruct, $datadir, $yymm ) ; $io->setFH( $filename, $ext ) ; $bulletin->print( $cChars ) ; } } } $io->inProgress( $filename, "close" ) ; $io->Close( $ext ) ; exit( 0 ) ; }

Documentation for "ldmConnect" The perl decoder "ldmConnect" pieces together segmented zone forecasts or other segmented products with WMO segment identifiers PAA, PAB, etc. For more information about the segmentation of large products and the use of bulletin segment heading lines utilizing the Pxx Indicator Group, see http://www.nws.noaa.gov/tg/bbb.html The "ldmConnect" script orders and combines bulletin segments in memory, strips out extra headers, and then writes the completed bulletin. If "ldmConnect" exits, it writes an "inProgress" file to maintain the current state of processing. When restarted, "ldmConnect" reads the "inProgress" file first. Non split reports are written directly. Usage with input on STDIN: ldmConnect [-a] [-s] [-l] [-e ext ] [yymm] [datadir] -a locates bulletin in directory structure "station ID"/"bulletin ID"/ -s strips out control characters (other than newline) -l turns on logging to create log files of the form "connectLog.<PID>.wmo" -e sets the file extension, default is "_for.wmo". File format is: yymmddhh + extension. yymm is the current year and month. Current yymm is default. datadir is the output directory for data, inProgress, and log files. Typically "ldmConnect" is located in the decoders directory. Typical pqact.conf entry: # Forecasts in yymmddhh_for.wmo # perl script ldmConnect, joins split bulletins, writes all bulletins DDS|PPS ^F[PTXEQWZ].... .... ([0-3][0-9]) PIPE /usr/local/ldm/decoders/ldmConnect (\1:yy)(\1:mm) data/forecasts