Joey, > Thanks so much for the input and answering these questions. NCEP is > tinkering with the idea of using the ldm to feed NCDC a supply of NOMADS > data. Data quantities could approach 300 GB/day to begin with about a > 10% increase per year. Most of the 300 GB/day would occur over a 3 hour > time span 4 times per/day during GFS and GENS time (25-30 GB/hour). At > the current time, we are just attempting to evaluate what methodologies > are possible with budgeting/hardware/network requirements to come later. > > It sounds like the ldm is a good candidate given the adequate > hardware/memory solution. 300 GB/d is a greater data-rate than any LDM is currently receiving. If the network bandwidth is available, however, I see no theoretical obstacles. What is the available bandwidth between NCDC and the source of the data? With that much data, you'll probably want the data-products to be on the large side (i.e., on the order of a megabyte or greater). I would also recommend using the recently-added "-i" option of the "pqinsert" utility when creating the data-products because it greatly reduces the load on the CPU. The LDM package has the capability of building a test version that can determine the maximum throughput possible. The upstream LDM uses a faux product-queue which auto-generates data-products of a specifiable size at a specifiable rate. Couple that with the existing "rtstats" reporting system and you should be able to quickly determine the feasibility of using the LDM. Please contact us if you have any questions. Regards, Steve Emmerson Ticket Details =================== Ticket ID: SYM-524686 Department: Support LDM Priority: Normal Status: On Hold
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.