[ldm-users] user-stories for the LDM replacement

[Mark Tucker]

I've been relatively quiet on these lists over the years but I thought I 
would add my two cents to the discussion as well.  I've been solely 
responsible for our ldm here at LSC for at least 8 years.

If a replacement is just going to make improvements to the existing 
"fetch it all" way of distributing data, please don't do it.  Over the 
past few years the ldm has been very reliable.  I don't even think about 
our ldm on a day to day basis any more.  It was not like this many years 
ago.  I do not wish to go back to the days of fighting with our ldm to 
stay up and keep a consistent data flow coming in.

I read over the wiki page a little this morning just to get a better 
sense of the scope that we're talking about here.

I guess I'm thinking a bit beyond the ldm as it is now. Some background: 
We purchased a NOAAport system several years ago because the routing 
beyond our campus was so bad that we could not receive our IDD feeds 
reliably.  I would guess that we only use a very small portion of that 
data feed on a regular basis.  The problem is that we use many, many 
other products on an irregular basis depending on weather events around 
the country or world.  This will also vary depending on the needs of our 
courses as they progress through the semesters.  Overall, we probably 
use less than 20% of our data on a daily basis but likely our needs 
would span to closer to 80% or 90% when evaluated on a product/category 
view spread over a semester or calendar year.  The end result is that we 
end up ingesting a HUGE volume of data, processing it in multiple 
different formats to satisfy the needs of various applications (ex: 
gempak vs. mcidas), and then filing it on disk only to scour it out a 
couple of days later when, most of the time, the majority was never used 
at all.

I feel that the fetch on demand way of doing things with mcidas and the 
IDV (ADDE) are the better approach.  In their current state there is a 
bit of room for improvement (IMO) but their approach to data access is 
one that can scale better in the long run.  With resolution increases in 
imagery, radar and forecast grid products the bandwidth demands are 
likely going to increase at a rate faster than our internet routes can 
support if we continue with the "fetch it all" approach.

I've thought about the "ldm" becoming something minimal like a caching 
proxy for ADDE requests that pulls in data when requested by our clients 
regardless of the client.  Not every site should need to run a full LDM 
server just because they'd like to view data in one of the gempak 
applications.  (Of course this doesn't address many other issues such as 
how we would handle backwards compatibility with things like the huge 
investment of time we have made in gempak scripts.)

I would also like to see some kind of catalog or index that improves our 
ability to find and access new products.

I'm sorry if I've rambled a bit around the topic with this.


Gilbert Sebenste wrote:
> On Sat, 29 Mar 2008, Daryl Herzmann wrote:
> 
>> I'll bite on this with my 2 cents!
>>
>> I was sort of surprised by this announcement and very much concerned.
>> Without a doubt, LDM has been the most stable and most important piece of
>> middle ware software I use.  Outside of the IDD, it is a key part of my
>> content generation and distribution system.  It just works and it is
>> something I don't worry about failing, since it rarely/never does!
>>
>> After a cursory look over the pages, it is not clear to me what is going
>> to happen with LDM.  Is this 'replacement' going to be written in Java? Is
>> it going to maintain RPC compatibility with LDM?  Aren't there other
>> projects doing some of the things mentioned, like Thredds Data Server(?),
>> etc?
>>
>> Talking with other LDM users this weekend, they expressed similar
>> concerns. I would much rather see some of those features go into the LDM
>> codebase without such a drastic shift to different / unproven technology.
>>
>> This new project looks very cool, but lets not replace LDM :)
> 
> +1 here. The current LDM works great, and I'm not sure reinventing the 
> wheel here is going to help. Unless it is going to work with older LDM's,
> I'm quite content with how this generation of LDM works. And this from 
> someone who people call "Captain Upgrade" for being bleeding edge!
> 
> *******************************************************************************
> Gilbert Sebenste                                                     ********
> (My opinions only!)                                                  ******
> Staff Meteorologist, Northern Illinois University                      ****
> E-mail: sebenste at weather.admin.niu.edu                                  ***
> web: http://weather.admin.niu.edu                                      **
> *******************************************************************************
> _______________________________________________
> ldm-users mailing list
> ldm-users at unidata.ucar.edu
> For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/ 


-- 
Mark Tucker
Meteorology Dept. Systems Administrator 

Lyndon State College 

http://apollo.lsc.vsc.edu 

mark.tucker at lyndonstate.edu 

(802)-626-6328