[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20050905: direction of NetCDF decoders / time_obs vs. time_nominal / patch for buoy2nc

On Mon, 5 Sep 2005, Unidata Support wrote:

>
> ------- Forwarded Message
>
> >To: address@hidden
> >From: Rob Cermak <address@hidden>
> >Subject: direction of NetCDF decoders / time_obs vs. time_nominal / patch 
> >for buoy2nc
> >Organization: University of Alaska-Fairbanks
> >Keywords: 200509050516.j855GOjo018952 netCDF decoders Perl
>
> Hi,
>
> I was wondering if perl was going to continue to exist for awhile in
> decoderland?
>
> I'm digging in a little bit on the decoders again, but didn't want
> to spend a whole lot of time if there were plans to migrate all the
> decoders to java in a year?


Rob,

the buoy2nc will be converted over to java, but when who know. currently i
been working on the grib 1 & 2 java decoder for the THREDDS data server
and have been getting many requests for mods for different types of files.
the buoy, synoptic, and upperair decoders will have changes made to them
that are similar to the metar2nc.new1. ie more CF compliant and probably
one file per day. the file naming will also change to
SURFACE_METAR_yyyymmdd_hhmm.nc, SURFACE_BUOY_yyyymmdd_hhmm.nc,
SURFACE_SYNOPTIC_yyyymmdd_hhmm.nc, and UPPERAIR__yyyymmdd_hhmm.nc
probably other changes too
>
> The one enhancement I have planned on the buoy2nc is to allow capture of
> multiple observations in a single hour from a single station.  Single
> hourly records are kept by a key: yyyymmddHH, this can be enhanced to
> capture the minute using reports that give us HHMM (key: yyyymmddHHMM).
> See more below.

i'll look at your work and try to accommodate you changes.

robb...

>
> I have also made a small change to allow reprocessing of past
> observations.  As it stands now, the program only allows processing on
> items no more than a day from the system time.  Rightfully so, since
> bulltins do not carry a full date, it needs all the help it can get.
>
> This small change allows the user to specify the year/month and override
> the current day/hour of the system.  If I wanted to reprocess data from
> 2005-08-28 19Z:
>
> ./buoy2nc -v /home/ldm/etc/buoy.cdl . 200508 28 19
>
> Might be better to test for a length of 4 vs. 8?
>   [yyyymm] vs. [yyyymmHHMM]
>
> --- /home/ldm/decoders/bin/buoy2nc      2005-05-26 15:19:40.000000000 -0800
> +++ ./buoy2nc   2005-09-04 07:56:13.000000000 -0800
> @@ -27,8 +27,14 @@
>         $cdlfile = $ARGV[ 0 ] ;
>         $datadir = $ARGV[ 1 ] ;
>         $yyyymm = $ARGV[ 2 ] ;
> +} elsif( $#ARGV == 4 ) {
> +       $cdlfile = $ARGV[ 0 ] ;
> +       $datadir = $ARGV[ 1 ] ;
> +       $yyyymm = $ARGV[ 2 ] ;
> +       $cday = $ARGV[ 3 ] ;
> +       $chour = $ARGV[ 4 ] ;
>  } else {
> -       die "Usage with input STDIN: buoy2nc [-v] cdlfile [datadir] 
> [yyyymm]\n";
> +       die "Usage with input STDIN: buoy2nc [-v] cdlfile [datadir] [yyyymm] 
> [cday] [chour]\n";
>  }
>
>  # check for cdl and netCDF ncgen
> @@ -217,8 +223,8 @@
>         next unless ( $bday && defined( $bhour ) && defined( $bmin ) ) ;
>         next if( $bmin > 59 || $bhour > 23 || $bday > 31 ) ;
>         # check for valid bulletin times against current time
> -       $cday = (gmtime())[ 3 ] ;
> -       $chour = (gmtime())[ 2 ] ;
> +       $cday = (gmtime())[ 3 ] if ($cday eq "") ;
> +       $chour = (gmtime())[ 2 ] if ($chour eq "") ;
>         # skip bulletins over 24 hours old or in the future
>         if( $bday == $cday ) {
>                 next if( $bhour > $chour ) ;
>
> Continued from above.   Patch below with test files.
>
> Things that changed:
>  * rpt_hash key is yyyymmddhhmm, however, the netcdf file key is
>    still yyyymmddhh based.
>  * time_obs/time_nominal is now always set for BBXX, time was
>    missing when there was a ship observation.
>  * time_obs/time_nominal updated for more precise observation
>  * time_obs has the most exact time, if available
>  * time_nominal has a rounded time
>  * added arguments to override current system time for reprocessing
>  * buoy.cdl: ship_call_len = 5->7, was dropping a lot of ships
>  * more chomping on headers
>  * clear variables at top of BBXX report loop.  On parse failure, they
>    should be erased, instead next is used leaving potential contamination
>    for future records.
>  * Don't accept ship IDs of 'SHIP'
>  * Add print ship in printvars
>  * One fix to prevent Pwa from being assigned //
>
> Added features:
>  * Allows for reprocessing
>  * Allows storage of multiple observations in a single hour.  More
>    platforms are reporting more than once an hour.
>
> This introduces a fundamental change to the storage of time.  These
> changes allow us to populate a relational database from the netcdf files.
>
> Please let me know what you roll into the new release.  If you have
> questions on anything, please let me know.
>
> Keep me posted on any minor releases of the decoders.  I was kinda
> surprised when 3.1.2 was out :)  AOOS will be starting to make heavy use
> of them and I'll be defintely sending more updates.
>
> Files:
>   test data : http://ak.aoos.org/ldm2/buoy_dat.tgz
>   script    : http://ak.aoos.org/ldm2/runb2nc
>   patch     : http://ak.aoos.org/ldm2/buoy2nc_3.1.2.diff.txt
>
> Rob
> --
> Rob Cermak : 907-474-7948 : FAX 907-474-7204 : PGP = 0x75869A6E
> Alaska Ocean Observing System : Data Management
> School of Fisheries and Ocean Sciences : University of Alaska Fairbanks
> address@hidden : address@hidden
> address@hidden : address@hidden
>
>
>
>
>
>
> --
> NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publicly available
> through the web.  If you do not want to have your interactions made
> available in this way, you must let us know in each email you send to us.
>
> ------- End of Forwarded Message
>

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================