[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: query of community interest in decoder technology



On Thu, 27 Oct 2005, Rob Cermak wrote:

> Hi,
>
> I just wanted to put out a general query to the community to see if you
> were looking for any additional decoders or would like to offer your
> services to help generate additional decoders.
>
> Due to the data needs of our Alaska Ocean Observing System, I've been
> developing decoders for CMAN and SHEF encoded messages streaming on the
> IDD.  A recent meeting indicated that these decoders may be useful to
> other groups in the ocean observing and grid computing areas.

Rob,

sorry for the delay in response, yes Unidata is interested in decoding the
CMAN and SHEF messages. Currently, i have my hands full doing other work
so efforts in this direction are taking a back seat. at lunch the other
day we talked about possibly adding the CMAN reports to the reports being
stored in the buoy data file. also, i'm working on the new netCDF file
format for station data,  similar to the metar.cdl.new1 because it is
closer to the CF data standards. maybe we could work together on this
task, it seems that you have an idea of how to decode these formats and i
have a good understanding to the ending file structure.

thanks,
robb...




>
> The CMAN decoder is fairly robust in that it functions much like the
> buoy2nc decoder but still may need to have some bugs ironed out.
>
> The SHEF decoder is really just starting up.  SHEF 2 has 3 main formats of
> messages: .A .B .E
>
> .A and .E are easy in that the message is very structured.
> .B is a free format setup by the data provider
>
> I have .A in a functional state, but I have broken the NetCDF storage
> portion of the code.  I'm not a expert in debugging perl/NetCDF.
> However, the code currently writes data into our Postgresql database using
> perl/DBI.  We went the perl/DBI route since we needed access to this
> information in to our system in the short term.
>
> Brief explaination of SHEF:
>
> .A is one or more parameters per specified time
> .E is one parameter over a period of specified time
> .B is free form, but we can attempt to build XML descriptions
>   that attempt to parse the message into its parts.  Crosswalker
>   from free-form to NetCDF or perl/DBI.
>
> Current SHEF 2 specification:
>   http://www.weather.gov/oh/hrl/shef/indexshef.htm
>
> SHEF messages include data from various instruments:
> USGS, BLM, RAWS and NWLON and doesn't stop there...
>
> Building a decoder is one step of the process.  As I've worked with
> the current decoders and these new ones, there are large hurdles at
> times and may or may never be fully addressed.
>
> Some of these issues are:
> * Format is not strictly followed.  (EX: METAR)
> * Time is not fully defined in the coded message and requires
>   inferral by the bulletin time or using what shreds of time
>   are in the coded message
> * Time zone handling/issues - real time decoding vs historical
>     reprocessing
> * Metadata/station listings
> * Overlapping networks (NDBC & CMAN & NOS; USGS/RAWS)
> * Various degress of QA/QC
> * Units! (touches on all coded messages)
> * Many others...
>
> I would be very interested in basic decoders that would allow other
> information such as satellite and radar to become available via NetCDF or
> RDBMs so that it could be served via OGC/OpenDap protocols via a GIS
> engine such as Mapserver.
>
> I'm interested in all feedback from the community and Unidata.  The
> attempt here is to energize the community rather than try and create
> more work for Unidata.
>
> I'm willing to become a focal point for the community as an interface to
> Unidata.  What tasks may be involved is not fully understood at this time.
> The Ocean Observing System is also undergoing some evoloution, it seems
> important now to try and forge some bridges between communities.
>
> Looking forward to your comments.
> Rob
> --
> Rob Cermak : 907-474-7948 : FAX 907-474-7204 : PGP = 0x75869A6E
> Alaska Ocean Observing System : Data Management
> School of Fisheries and Ocean Sciences : University of Alaska Fairbanks
> address@hidden : address@hidden
> address@hidden : address@hidden
>
>
>

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================