distributing mesonet data for use in IDV

[John Horel]

Robb-

Thanks, that gets me pointed in the right direction. We have 1 database, 
but separate tables for access through different types of queries. We 
have a "latest" table that serves up the most current ob at all 15000 
stations for really fast queries. Alternatively, we have the complete 
set of tables for more generalized queries that allows people to access 
any data in the past 10 years. Those are in monthly tables and 
subdivided into groupings of variables (basic, precip, road wx, etc.). 
So, we'd support both paradigms via IDV for sure. All the metars are in 
our tables (including the raw report), so if you would like to 
experiment at some point with grabbing data from here, let me know.

Regards, John


Robb Kambic wrote:
> On Wed, 14 Feb 2007, John Horel wrote:
> 
>> Robb-
>>
>> Starting to put together the equipment proposal to Unidata, which in 
>> part, would include a data server to serve up mesonet observations. 
>> So, I'd like to follow up on our conversations in San Antonio as far 
>> as possible approaches. I'm still a bit fuzzy as to what would be the 
>> best way to do it. My preference would be to develop a query directly 
>> to our database rather than having to store the data in an external 
>> hosted format. That way we don't have to keep around all 10 years of 
>> data in a netCDF file format or some such. Have you had a chance to 
>> proceed with your middleware to handle metars? And, could you describe 
>> that approach a bit more for me?
>>
>> Regards,
>>
>> John
>>
> 
> John,
> 
> your timing is perfect, yesterday our group had a meeting on this.  in 
> fact, the last month i created a realtime database to store the Metars. 
> at this time there is a simple url servlet interface that permits one to 
> make queries against it.  so your idea of maintaining the data in a 
> database is the correct approach. i'm just starting on making a general 
> station observation dataset adapter that would create the link from some 
> data repository ie database into the Java netcdf library, then the IDV. 
> the station observation dataset adapter would get enough info so it 
> could know how to query the proper data repository. at this time, i 
> don't know the type/amount of info needed. i'll keep you informed on our 
> progress.
> 
> do you keep all the data in one database?  i was planning on daily 
> creating a new database file for data that was 3 days old and only 
> keeping ~3 days of data in the realtime database. this way the 
> performance would be good for the realtime requests and archive requests 
> would require opening the daily database files.
> 
> Machine requirements guess...
> the Thredds Data Server would have to be installed on the machine with 
> cpu power to handle the amount of requests plus good network 
> conectivity. i'm sure you have a better handle on the disk space 
> requirements. you might want to look at THREDDS page for requirements 
> for the TDS.
> 
> http://www.unidata.ucar.edu/projects/THREDDS/
> 
> robb...
> 
> =============================================================================== 
> 
> Robb Kambic                       Unidata Program Center
> Software Engineer III               Univ. Corp for Atmospheric Research
> rkambic at unidata.ucar.edu           WWW: http://www.unidata.ucar.edu/
> =============================================================================== 
>