[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #QCL-983945]: size of products in product queue

Hi Jerry,

re: product queues used in TIGGE testing were each at least 12 GB
> 12 GB!  Wow ... do you still recommend that the server has enough memory to 
> memory
> map that file?

It is best if the entire product queue can be memory mapped as a whole.  If it
can not, individual products are memory mapped, so there is no absolute rule
that states that one _must_ have enough RAM to memory map the entire queue.
It is just more efficient that way.

> A 4 Gb product queue is the largest I have right now.

The size that the queue needs to be is a function of how much data you want to
send downstream, the rate that it is to be sent downstream, and how slow
the slowest downstream link is.

re: need good, high speed network connectivity
> Most of the internal stuff will be over Gigabit network.

Excellent.  

> Not so outside of our network, but
> anything going outside the center will probably be smaller products anyway.

Right.  It is not necessarily the size of the products that are important.  It
is the volume rate of the stream, and volume is the product of size and 
frequency
of insertion.

re:  How many files are we talking about? 
> These are passes from the two EOS satellites AQUA and TERRA.  We get between 3
> and 6 passes each per day.
> see http://eosdb.ssec.wisc.edu/modisdirect/
> 
> We are primarily interested in distributing the level 2 hdf products which 
> are much smaller.  There
> may be a half dozen or so per pass
> 
> E.g.
> ftp://aqua.ssec.wisc.edu/pub/aqua/modis/2006_06_07_158/products
> 
> All of which may not be inserted.

Sounds more than doable.

> We may also internally want to move the 1km level-1 passes and possibly the 
> half and quarter km
> level-1 passes.  There are usually 3 of these per pass during the day.  At 
> night we have only the 1
> km level-1 hdf.  These are the big guys,  but usually less than 400 MB when 
> compressed.
> 
> E.g.
> ftp://aqua.ssec.wisc.edu/pub/aqua/modis/2006_06_07_158/

You will have no problems with internal distribution.

re: create product IDs that allow flexibility for the end users
> Understood.   I am already pushing some netcdf files with MODIS products
> created with McIDAS to a local awips machine, and some NWS awips machines. 
> The product description naming convention used was
> 
> E.g.
> SSEC_AWIPS_MODIS_4KM_Visible_AQUA_20060428_1900.7300
> 
> (underscore deliminated)
> Source Center
> Stream type
> Instrument
> Resolution
> Product description
> Satellite source
> YYYYMMdd
> HHMM
> product number id
> 
> I used "_" as a field separator, and '.' before the product ID number.
> 
> Would spaces be better?

No, underscores are fine.  I personally would use spaces, but this is a personal
preference.  Matthew's preference for the Antarctic-IDD was periods; yours is
underscores; mine is spaces.  All work fine.

> I haven't thought too hard about the naming convention for these new products,
> but I was planning to use some of the same types of information.

Creating useful product IDs is perhaps the hardest part of the exercise.  One
of the reasons for this is that once you start using one convention, it is hard
to change since end users will have spent time and effort creating pqact.conf
actions that work, and they will _not_ want to change (believe me on this one 
:-)

> Thanks Tons Tom!

No worries.  I will be very interested in your comments as you create this 'IDD'
network.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: QCL-983945
Department: Support IDD
Priority: Normal
Status: Closed