[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: distributing mesonet data for use in IDV



Robb-

Thanks, that gets me pointed in the right direction. We have 1 database, but separate tables for access through different types of queries. We have a "latest" table that serves up the most current ob at all 15000 stations for really fast queries. Alternatively, we have the complete set of tables for more generalized queries that allows people to access any data in the past 10 years. Those are in monthly tables and subdivided into groupings of variables (basic, precip, road wx, etc.). So, we'd support both paradigms via IDV for sure. All the metars are in our tables (including the raw report), so if you would like to experiment at some point with grabbing data from here, let me know.

Regards, John


Robb Kambic wrote:
On Wed, 14 Feb 2007, John Horel wrote:

Robb-

Starting to put together the equipment proposal to Unidata, which in part, would include a data server to serve up mesonet observations. So, I'd like to follow up on our conversations in San Antonio as far as possible approaches. I'm still a bit fuzzy as to what would be the best way to do it. My preference would be to develop a query directly to our database rather than having to store the data in an external hosted format. That way we don't have to keep around all 10 years of data in a netCDF file format or some such. Have you had a chance to proceed with your middleware to handle metars? And, could you describe that approach a bit more for me?

Regards,

John


John,

your timing is perfect, yesterday our group had a meeting on this. in fact, the last month i created a realtime database to store the Metars. at this time there is a simple url servlet interface that permits one to make queries against it. so your idea of maintaining the data in a database is the correct approach. i'm just starting on making a general station observation dataset adapter that would create the link from some data repository ie database into the Java netcdf library, then the IDV. the station observation dataset adapter would get enough info so it could know how to query the proper data repository. at this time, i don't know the type/amount of info needed. i'll keep you informed on our progress.

do you keep all the data in one database? i was planning on daily creating a new database file for data that was 3 days old and only keeping ~3 days of data in the realtime database. this way the performance would be good for the realtime requests and archive requests would require opening the daily database files.

Machine requirements guess...
the Thredds Data Server would have to be installed on the machine with cpu power to handle the amount of requests plus good network conectivity. i'm sure you have a better handle on the disk space requirements. you might want to look at THREDDS page for requirements for the TDS.

http://www.unidata.ucar.edu/projects/THREDDS/

robb...

===============================================================================
Robb Kambic                       Unidata Program Center
Software Engineer III               Univ. Corp for Atmospheric Research
address@hidden           WWW: http://www.unidata.ucar.edu/
===============================================================================