[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20021001: UFRJ as a primary IDD redistribution point for Brazil (cont.)



>From: David Garrana Coelho <address@hidden>
>Organization:  UFRJ
>Keywords:  200209271806.g8RI6v115886 IDD Brazil Meteoforum

Hi David,

I am sorry that it has take me so long to get back to you on your
generous offer to become a top level IDD site for Brazil.  I was out of
town at the McIDAS Users' Group meeting in Madison, WI where I was too
busy to respond fully.

re: likely to need to increase LDM queue size to 500 MB or larger
>Thats ok, i can do it.

Very good.

re: what OS are you running
>RedHat 7.3 full patched

OK.

re: upgrade your LDM to 5.2.1
>I will do it asap, will report to you when done.

OK.  Here is where I need to let you know what we have been doing
to the LDM in the past few weeks.

We have been busily working on a new release of the LDM that works
better for long haul distribution of data.  We are currently testing
this distribution, LDM v5.3, and are enthusiastic with our results.
Even though this version is not yet released, and even though this
version can not be used to feed data _to_ older LDM versions (i.e., it
is not yet backwardly compatible), I want you to upgrade to it.  5.3
can receive data from non-5.3 LDMs, but it can't send data to non-5.3
LDMs.  The LDM at UFPA is already running v5.3 (I install our latest
test versions at UFPA as soon as we want to run long haul experiments),
so you will be able to feed them with no problems.

You can grab LDM v5.3.0 using anonymous FTP from our FTP server
ftp.unidata.ucar.edu:

machine:   ftp.unidata.ucar.edu
directory: pub/ldm5/test
file:      ldm-5.3.0.tar.Z

re:
2) change the data requests in your ldmd.conf to:

   o split the IDS+DDPLUS feed into thirds (better insures that the
     products get their with little to no latency)

   o add ingest of the Unidata-Wisconsin (LDM 5.1.x MCIDAS or LDM 5.2+
     UNIWISC feedtype).  The current GOES-East (GOES-8) images in this
     datastream are useful for northern Brazil.  We will be adding
     sectors tailored to Brazil in the coming weeks.

   o add/modify ingestion of global model data (the non-global model
     data available in our HDS stream is useful for the continental
     US, Canada, Mexico, and the Caribbean only).

   o change the host from which you are receiving data (we will switch
     you to a machine we maintain at the National Science Foundation
     in Washington, D.C.

   o add the reporting of Realtime Statistics (available with LDM 5.2+)
     in addition to, and eventually replacement of 'pqbinstats'.

   o we may be asking you to upgrade your LDM somewhat frequently
     as we are focusing a lot of effort on improving its design
     especially for long distance data delivery (like to South America).

>I need more specific instructions (a.k.a. step by step) to do this.

I want you to have, at a minimum, the following entries in your ldmd.conf
file:

ldmd.conf -- begin

###############################################################################
# Exec Entries
###############################################################################
#
# Programs that can share a queue with rpc.ldmd server, these programs
# are started by the "exec" command and are in the same process group.
#
exec    "pqbinstats"
exec    "pqact"
exec    "rtstats -h rtstats.unidata.ucar.edu"
#exec   "pqsurf"
#
###############################################################################
# Request Entries
###############################################################################
#
# LDM5 servers request data from Data Sources
#
#       request <feedset> <pattern> <hostname pattern>
#
request DDPLUS|IDS      "^[^S]" atm.geo.nsf.gov
request DDPLUS|IDS      "^S[AR]" atm.geo.nsf.gov
request DDPLUS|IDS      "^S[^AR]" atm.geo.nsf.gov
request HDS             "^H.[I-P]... KWB[^K]" shemp.unidata.ucar.edu
request HDS             "^H[A-Z][ABCD][A-Z][0-9][0-9] KWB." 
shemp.unidata.ucar.edu
request HDS             ".*(ECMWF|EGRR)" shemp.unidata.ucar.edu
request UNIWISC         ".*" shemp.unidata.ucar.edu
#
###############################################################################
# Allow Entries
###############################################################################
#
# Giving permission for a Data Sink to perform a request to your LDM
#
# allow <feedset> <hostname pattern>
#
# Giving permission to your own machine and Unidata
#
# Under no circumstances comment out the next allow entry to localhost
# The LDM will NOT start if the lines are commented out.
allow   ANY
        
^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$))

ldmd.conf -- end

The things that are new are:

1) an 'exec' entry for rtstats.  This is a new statistics reporting facility
   in LDM 5.2.x

2) three request lines for 'DDPLUS|IDS'.  These split the feed of this
   stream into thirds (each entry will deliver roughly one third of the
   number of products in the stream).  The DDPLUS|IDS stream (this is
   actually one stream not two) contains almost all of the observational
   data that is on the global telecommunications gateway (e.g., global
   surface, upper air, ship/buoy, etc. observations).

3) three request lines for 'HDS'.  These are the global grids that are
   available in the NOAAPORT broadcast that feed the US NWS forecast
   offices.  The global grids are currently ones from NCEP, the UK Met.
   Office, and some from the ECMWF.

4) feed of all satellite images in the 'UNIWISC' stream (formerly referred
   to as the 'MCIDAS' stream).

5) make sure to give LDM access (ldmping, notifyme, feedme) to any machine
   here in our offices.  This helps us to monitor your LDM operation.

>I
>configured our LDM/GEMPAK based on countless tries and sucess/failures. In
>fact, we are still unable to get/decode aircraft data because this method
>didn't worked with this kind of data.

The ACARS data you are referring to is only available from a
point-to-point LDM feed from the Forecast Systems Laboratory (FSL) of
NOAA.  I am not sure if they would be willing to feed this data non-US
entities.  Besides that, given the unhealthy financial condition of the
airlines that are providing the data, there is a real possibility that
we will all lose access to the data.

>Our ldmd.conf and pqact.conf is
>loosely based on versions of these files found in the web. We believe that
>exist other feeds than the ones we decode/keep that would be useful to us,
>but we (i) only sucessfully decoded/know of the ones you see in our conf
>files.

OK.  I will work with you to identify what kinds of data are available
in which feeds so that you can take full advantage of the IDD.

>Thats ok, but some kind of summary/description of all the data we will be
>receiving would be great. So we and other users could choose easily what
>to keep and what not to keep.

A detailed description of exactly what data is contained in each product
sent in the IDD is beyond the scope of email exchanges.  I will try
and point you to web sites that list the kinds of data that is available
in future emails.

re: adding more memory and disk is always useful, especially on Linux

>We will consider doing this in near future. We have plenty of disk space
>in other machines, so the data we keep locally wont take space for very
>long in the machine LDM is running.

Sounds good.

re: giving me access to the machine at UFRJ that is running the LDM

>We can give you access to the account where ldm is installed, so you can
>do the inital setup and everything,

Wonderful!  Thank you!

>but we'd like to have a descripition
>of what changes have been done to the installation and configuration and
>how it has been done, so we can help you in this administrative stuff in
>the future.

Absolutely.  I always do this.  After all, our objective is for UFRJ to
become the top level IDD site for Brazil (and possibly Argentina).  The
only way that you can do this is by totally understanding how the LDM
works.

>I cannot give you root access because some data stored in our
>machines in our local network are under a NDA (Non-Disclosure Agreement),
>so giving this kind of access to any machine outside our network is not
>(legally) possible at moment.

This is not a problem.  What I will do is send you emails with
instructions on what things need to be done by 'root' on your machine.
It will slow the upgrade process (not a big deal), but I totally
understand your constraints.

>But you can give me the instructions and i
>will make the modifications to configuration files and activate the
>necessary deamons.

The outline for building and installing the LDM can be found in:

Unidata HomePage
http://my.unidata.ucar.edu

  Software
  http://my.unidata.ucar.edu/content/software

    LDM
    http://my.unidata.ucar.edu/content/software/ldm/index.html

From the last page, you should review:

LDM Preinstallation Requirements
http://my.unidata.ucar.edu/content/software/ldm/ldmPreInstallList.html

and

LDM Source Installation Check List
http://my.unidata.ucar.edu/content/software/ldm/ldmSourceInstallList.html

>I believe some kind of private/personal/temporary e-mail adress of yours
>is recommended so i can send the login/password to you more safely.

My Unidata email address is:  address@hidden

Again, I look forward to working with you and your UFRJ collegues on the
establishment of what I hope will be a service used throughout Brazil!

Tom