Re: [gembud] Using DCMETR

Thanks to everyone who helped me with this issue, I am going to pass along
my solution in case it should ever serve to help someone in the same
situation. Thanks also to Steve Chiswell who didn't play a direct part in
this discussion, but who's previous assistance to other user's did me good.

Here is the script I wrote. I iterate over the HHZ.TXT files off of the
tgftp site and pull only the US stations, since that's all I am concerned
with. I prepend the appropriate bulletin information (AFOS formatting) and
then save it to a temporary file. I then feed the temporary file into
dcmetr, which then spits out the correctly formatted GEMPAK file. The
observation file I am currently passing in as an argument. I also remove the
previously created temporary file before writing a new one, although I
suppose this could be part of a cleanup step at the end of the script. But
hey, whatever gets the job done right?

#!/bin/csh -f
set obsFile=$1
set currentLine=2
set totalLines=`cat ${obsFile} | wc -l`
set currentDate_date=`date -u +%y%m%d`
set currentDate_time=`date -u +%H%M`
echo "Reading through ${obsFile}"
echo "Total of ${totalLines} lines. Standby."
rm latest.txt
set stationCount=0
printf "" > latest.txt
while (${currentLine} <= ${totalLines})
    set firstChar=`sed -n ${currentLine}p ${obsFile} | cut -c1-1`
    if (${firstChar} == "K") then
        printf "ZCZC\n\n" >> latest.txt
        sed -n ${currentLine}p ${obsFile} >> latest.txt
        printf "\n\nNNNN\n\n" >> latest.txt
        set currentLine=`expr ${currentLine} + 3`
        set stationCount=`expr ${stationCount} + 1`
        echo "Read a station, line ${currentLine}."
    else
        set currentLine=`expr ${currentLine} + 3`
    endif
end
echo "Total stations read = ${stationCount}."
echo "Now decoding with dcmetr."
dcmetr -c ${currentDate_date}/${currentDate_time} -v 4 -d - YYYYMMDDHH.metar
< latest.txt



On Thu, Apr 15, 2010 at 3:03 PM, <pmanousos@xxxxxxxxxxxxxxxxxxx> wrote:

>
> Rats Colton - I forgot to tell you one other thing.  I read the entire file
> line by line and PREPEND the ob with a header line that contains the date
> time and other header info.  Here is the loop - sorry about that.  Pete
>
> set curline=1
> set maxlines=`cat $OBFILE | wc -l`
> while ($curline <= $maxlineS)
>         #read header line
>         set line=`sed -n ${curline}p $OBFILE`
>         # prepend header line with critical info for dcmetr
>         echo "SAUS80 KWBC "`echo $line | cut -c 9-10``echo $line | cut -c
> 12-13``echo $line | cut -c 15-16` > temp.ob
>         # now goto the next line and get the actual ob
>         @ curline = $curline + 1
>         set line=`sed -n ${curline}p $OBFILE`
>         echo "$line" >> temp.ob
>         # now run it through addafos.csh and dcmetr
>         cat temp.ob | ./addafos.csh | dcmetr -a 500 -v 2 -t 60
> $GEMDATA/hrly/YYYYMMDD.hrly
>         # we read two lines, so increase the counter by 2
>         @ curline = $curline + 2
> end
>
>
> SAUS80 KWBC
>
> Colton Ancell <colton.ancell@xxxxxxxxx> wrote on 04/15/2010 01:26:54 PM:
>
>
> > Pete and Jason, thanks. I am now getting somewhere,
> > although not completely.
>
> >
> > I added the header/footer control characters and am now
> > spitting out a log. I am also using the cycles that Pete
> > showed me instead of the data I was pulling before. The
> > logfile for dcmetr is telling me it's reading all the way
> > through the file, but tells me 0 bulletins were processed.
> > Below is a copy of the logfile and a copy of the commands
> > I'm running. My addafos.csh script is exactly like Pete's.
> >
> > My command:
> >
> > cat 16Z.TXT | ./addafos.csh | dcmetr -a 500 -v 2 -t 60
> > $GEMDATA/obs/hrly/YYYYMMDD.hrly
> >
> > Also tried:
> >
> > cat ob.txt | ./addafos.csh | dcmetr -a 500 -v 2 -t 60
> > $GEMDATA/obs/hrly/20100415.hrly
> >
> > dcmetr.log:
> >
> > [3532] 100415/1221[DC 3]  Starting up. Version 5.11.1
> > [3532] 100415/1221[DCMETR 7]  DCMETR version:  3.3
> > [3532] 100415/1221[DC 2]  read 11/102399 bytes strt 0 newstrt 11
> > [3532] 100415/1221[DC 2]  read 4096/102388 bytes strt 11
> > newstrt 4107
> > [3532] 100415/1221[DC 2]  read 4096/98292 bytes strt 4107
> > newstrt 8203
> > [3532] 100415/1221[DC 2]  read 4096/94196 bytes strt 8203
> > newstrt 12299
> > [3532] 100415/1221[DC 2]  read 4096/90100 bytes strt 12299
> > newstrt 16395
> > [3532] 100415/1221[DC 2]  read 4096/86004 bytes strt 16395
> > newstrt 20491
> > [3532] 100415/1221[DC 2]  read 4096/81908 bytes strt 20491
> > newstrt 24587
> > [3532] 100415/1221[DC 2]  read 4096/77812 bytes strt 24587
> > newstrt 28683
> > [3532] 100415/1221[DC 2]  read 4096/73716 bytes strt 28683
> > newstrt 32779
> > [3532] 100415/1221[DC 2]  read 4096/69620 bytes strt 32779
> > newstrt 36875
> > [3532] 100415/1221[DC 2]  read 4096/65524 bytes strt 36875
> > newstrt 40971
> > [3532] 100415/1221[DC 2]  read 4096/61428 bytes strt 40971
> > newstrt 45067
> > [3532] 100415/1221[DC 2]  read 4096/57332 bytes strt 45067
> > newstrt 49163
> > [3532] 100415/1221[DC 2]  read 4096/53236 bytes strt 49163
> > newstrt 53259
> > [3532] 100415/1221[DC 2]  read 4096/49140 bytes strt 53259
> > newstrt 57355
> > [3532] 100415/1221[DC 2]  read 4096/45044 bytes strt 57355
> > newstrt 61451
> > [3532] 100415/1221[DC 2]  read 4096/40948 bytes strt 61451
> > newstrt 65547
> > [3532] 100415/1221[DC 2]  read 4096/36852 bytes strt 65547
> > newstrt 69643
> > [3532] 100415/1221[DC 2]  read 4096/32756 bytes strt 69643
> > newstrt 73739
> > [3532] 100415/1221[DC 2]  read 4096/28660 bytes strt 73739
> > newstrt 77835
> > [3532] 100415/1221[DC 2]  read 4096/24564 bytes strt 77835
> > newstrt 81931
> > [3532] 100415/1221[DC 2]  read 4096/20468 bytes strt 81931
> > newstrt 86027
> > [3532] 100415/1221[DC 2]  read 4096/16372 bytes strt 86027
> > newstrt 90123
> > [3532] 100415/1221[DC 2]  read 4096/12276 bytes strt 90123
> > newstrt 94219
> > [3532] 100415/1221[DC 2]  read 4096/8180 bytes strt 94219
> > newstrt 98315
> > [3532] 100415/1221[DC 2]  read 4084/4084 bytes strt 98315
> > newstrt 102399
> > [3532] 100415/1221[DC 2]  read 0/0 bytes strt 102399 newstrt 102399
> > [3532] 100415/1221[DC -9]  End of input data file.
> > [3532] 100415/1221[DC 5]  Normal termination.
> > [3532] 100415/1221[DC 2]  Number of bulletins read and processed: 0
> > [3532] 100415/1221[DC 6]  Shutting down.
> >
> > So now it reads the file, but won't spit it out anywhere,
> > and does not seem to recognize the data as METAR bulletins.
> >
> > Thanks for everyone's help.
> >
> > P.S. I tried looking for Steve Chiswell's previous post on
> > this issue and could not seem to find it. Can anyone link
> > me to it? Thanks!
> >
>
> > On Thu, Apr 15, 2010 at 9:22 AM, <pmanousos@xxxxxxxxxxxxxxxxxxx
> > > wrote:
> >
> > Hi Colton - We are pulling data off NOAA's tgftp server
> > and running it through dcmetr.  But we found (thanks to
> > Steve Chizwell) that we needed to bound the ob with a
> > string of "AFOS" control characters for the decoder to
> > work.  Here is what we did -
> >
> > 1. Created a script "addafos.csh" as follows (these are
> > all zeroes, not capital "Oh"s)
> >
> > #!/bin/csh -f
> > printf "\001\r\r\n000 \r\r\n"
> > cat
> > printf "\r\r\n\003"
> >
> > 2. chmod 777 afos.csh)
> >
> > 3. Run the ob (contained in ob.txt) through addafos.csh
> > and dcmetr as follows
> > cat ob.txt | ./addafos.csh | dcmetr -a 500 -v 2 -t 60
> > $GEMDATA/obs/hrly/YYYYMMDD.hrly
> >
> >
> > Our decode script which contains step 3 actually will make
> > "addafos.csh" if it doesnt exist locally. By the way you
> > may want to try pulling ftp://tgftp.nws.noaa.
> > gov/data/observations/metar/cycles/hhZ.txt (where hh is
> > the current hour in UTC).  This contains specials and will
> > allow you decode all obs around the globe in less than 15
> > min.  If you run a master decode script every 15 min, you
> > will have updated obs (surface, not ship), from anywhere
> > over the globe.
> >
> > Pete
> >
> > gembud-bounces@xxxxxxxxxxxxxxxx wrote on 04/15/2010 03:19:33 AM:
> >
> >
> > > Hello all!
> > >
> > > I am having issues using the DCMETR decoder, and knew this
> > > was the right place to ask for help.
> > >
> > > As I am not a university or institution of note, I am not
> > > using LDM or the IDD, and instead had the idea of maybe
> > > pulling data off of the NOAA ftp server at ftp://tgftp.
> > nws.noaa.gov/
> > > . I can successfully login via scripting, etc. I can cd to the
> > > ftp://tgftp.nws.noaa.gov/SL.us008001/DF.an/DC.sflnd/DS.metar
> > > directory, which contains updated metar files. However,
> > > these are in a .txt format, and I'm not 100% sure if they
> > > are formatted the way DCMETR needs them to be as input
> > > files. I am rather new to this, but am learning quickly. I
> > > just need some data to play with! If anyone can tell me
> > > what I may be doing wrong, I would greatly appreciate it.
> > > Here are the commands I'm using once I have my file.
> > >
> > > dcmetr -c YYMMDD/HHNN sfcdata.gem < sn.0114.txt
> > >
> > > (the above command's general format I copied from the
> > > phelp file for dcmetr)
> > >
> > > dcmetr -c sfcdata.gem < sn.0114.txt
> > > dcmetr -c /home/gempak/NAWIPS/gempak/data/sfcdata.gem <
> > > /home/gempak/NAWIPS/gempak/data/sn.0114.txt
> > >
> > > As you can see, I try specifying explicitly and still, all
> > > dcmetr does is hang for about half a second, then put me
> > > back at the prompt. Nothing is generated as far as logs
> > > go, so I have no idea what's going on. I hope this is
> > > clear enough. Please help!
> > >
> > > --
> > > Colton Ancell
> > > Geophysics Major, Texas Tech University.
> > > KE5DXX_______________________________________________
> >
> > > gembud mailing list
> > > gembud@xxxxxxxxxxxxxxxx
> > > For list information or to unsubscribe,  visit: http:
> > > //www.unidata.ucar.edu/mailing_lists/
> > ----------------------------------------- The information
> > contained in this message is intended only for the
> > personal and confidential use of the recipient(s) named
> > above. If the reader of this message is not the intended
> > recipient or an agent responsible for delivering it to the
> > intended recipient, you are hereby notified that you have
> > received this document in error and that any review,
> > dissemination, distribution, or copying of this message is
> > strictly prohibited. If you have received this
> > communication in error, please notify us immediately, and
> > delete the original message.
> >
> >
> >
> > --
> > Colton Ancell
> > Geophysics Major
> > KE5DXX ----------------------------------------- The information
> contained in this message is intended only for the personal and confidential
> use of the recipient(s) named above. If the reader of this message is not
> the intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that you have received this
> document in error and that any review, dissemination, distribution, or
> copying of this message is strictly prohibited. If you have received this
> communication in error, please notify us immediately, and delete the
> original message.




-- 
Colton Ancell
Geophysics Major
KE5DXX