[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
20030715: IDS|DDPLUS latencies and building an LDM with large file support (cont.)
- Subject: 20030715: IDS|DDPLUS latencies and building an LDM with large file support (cont.)
- Date: Tue, 15 Jul 2003 13:29:07 -0600
>From: Celia Chen <address@hidden>
>Organization: NCAR/RAP
>Keywords: 200307151639.h6FGdSLd007355 LDM IDD rtstats largefile
Celia,
>I had
>exe "rtstats -q /home/ldm/data/ldm.pq -d /home/ldm/logs -h
>rtstats.unidata.ucar.edu"
>in chisel's ldmd.conf file as the last exec line. Is there a reason for the
>above line not working properly?
If you really did have 'exe' instead of 'exec', and the ldmd.conf
listing below shows that you did, then yes, this was the problem. 'exe'
is not a recognized abbreviation for 'exec'.
The -q, -d, and -h entries were OK, but not needed _if_ your LDM setup is
such that the queue is in ~ldm/data and the log flie directory is
~ldm/logs.
>However, I just replaced it with
> exec "rtstats -h rtstats.unidata.ucar.edu"
>hope this will send the statistics to you.
We are now getting real time stats from chisel as can be seen from:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+chisel.rap.ucar.edu
From the latency plot for IDS|DDPLUS, I can say that chisel is getting
the data from thelma with no delay. This tells me that if some METARs
are coming in late, they were late in the NOAAPORT broadcast; and
there is nothing we can do about that.
>PS - Below is chisel's ldmd.conf file:
OK, here are some suggestions for tuning your ldmd.conf file:
1) change the Exec Entries section to clean things up:
exec "pqbinstats"
exec "pqact -l /home/ldm/logs/pqact.log"
exec "pqsurf -p ^(SACN43|S[APX]|FXUS43) -Q /home/ldm/data/pqsurf.pq -l
logs/pqsurf.log etc/pqsurf.conf"
exec "rtstats -h rtstats.unidata.ucar.edu"
The suggested changes assume that the LDM queue can be found in
~ldm/data, and the log files are written into ~ldm/logs.
2) The NLDN feed is supposed to be available only by point-to-point feed
from SUNY Albany. This is the agreement the Unidata community has
with SUNY Albany. Breaching it could result in the community losing
this feed. We should, therefore, not be relaying this data to you from
the atm.geo.nsf.gov machine.
request NLDN ".*" striker.atmos.albany.edu PRIMARY
request NLDN ".*" atm.geo.nsf.gov ALTERNATE
On top of this, striker.atmos.albany.edu is the injection machine
for NLDN data. If it goes down, there will be no NLDN data from
any machine.
Given these two observations, please remove your NLDN request line that
points to atm.geo.nsf.gov. I will be removing the allow for NLDN
data from the ldmd.conf file on atm.
Also, you are allowing a number of machines to feed 'ANY' from chisel.
This is OK for RAP machines and mostly OK for other .ucar.edu machines,
but is not OK for others like the .gov machines. The same comment can
be made for WSI data. You should, therefore, change your allow lines
for non-RAP machines from:
allow ANY ...
to:
allow ANY-WSI-NLDN ...
3) The following suggestion will help cut down on the number of rpc.ldmd
connections to upstream feeds sites; they should have no effect
on your reception of data:
- combine your DIFAX, FSL2, and MCIDAS data requests to thelma and atm:
change the three lines:
request FSL2 ".*" thelma.ucar.edu PRIMARY
request DIFAX ".*" thelma.ucar.edu PRIMARY
request MCIDAS ".*" thelma.ucar.edu PRIMARY
and:
request FSL2 ".*" atm.geo.nsf.gov ALTERNATE
request DIFAX ".*" atm.geo.nsf.gov ALTERNATE
request MCIDAS ".*" atm.geo.nsf.gov ALTERNATE
to the single lines:
request DIFAX|FSL2|UNIWISC ".*" thelma.ucar.edu PRIMARY
request DIFAX|FSL2|UNIWISC ".*" atm.geo.nsf.gov ALTERNATE
(UNIWISC is the accepted name for Unidata-Wisconsin datasteam
products. MCIDAS still works, but it has been deprecated).
Cutting down on the number of rpc.ldmds running on both
your machine and the upstream feed host(s) will help both
machines run more efficiently.
4) remove the request for NEXRD2 data to atm.geo.nsf.gov. atm does not
receive this feed, so requesting the data from it does nothing more
than add a load to your machine AND to atm. Unfortunately, there
is no current failover machine for this feed.
5) I would normally ask you to remove the NMC2 (CONDUIT) feed to
atm since atm doesn't have this data, but we are trying to get
the weather service to feed atm. If/when this does happen, thelma
will ingest CONDUIT from atm. Having chisel ingest from thelma
and atm does not give you any real redundancy. Because of this,
I think you would do better to request the CONDUIT stream from
the University of Wisconsin/AOS machine.
change:
request NMC2 ".*" thelma.ucar.edu PRIMARY
request NMC2 ".*" atm.geo.nsf.gov ALTERNATE
to:
request CONDUIT ".*" thelma.ucar.edu PRIMARY
request CONDUIT ".*" f5.aos.wisc.edu ALTERNATE
At some point in the near future (not now), you might be asked to
change this to:
request CONDUIT "[09]$" f5.aos.wisc.edu PRIMARY
request CONDUIT "[18]$" f5.aos.wisc.edu PRIMARY
request CONDUIT "[27]$" f5.aos.wisc.edu PRIMARY
request CONDUIT "[36]$" f5.aos.wisc.edu PRIMARY
request CONDUIT "[45]$" f5.aos.wisc.edu PRIMARY
request CONDUIT ".*" atm.geo.nsf.gov ALTERNATE
You will need to contact Pete Pokrandt <address@hidden>
at UW/AOS to get him to add an allow for CONDUIT for chisel.
Combining the above recommendations, your cleaned-up ldmd.conf file would
look like:
###############################################################################
# Exec Entries
###############################################################################
#
exec "pqbinstats"
exec "pqact -l /home/ldm/logs/pqact.log"
exec "pqsurf -p ^(SACN43|S[APX]|FXUS43) -Q /home/ldm/data/pqsurf.pq -l
/home/ldm/logs/pqsurf.log etc/pqsurf.conf"
#
# rtstats uses the LDM to send product latency statistics to the UPC.
# We are currently evolving from pqbinstats to rtstats, but in the
# mean time we hope sites will run both. For more info on rtstats
# please see the man page.
exec "rtstats -h rtstats.unidata.ucar.edu"
#
###############################################################################
# Request Entries
###############################################################################
#
# LDM5 servers request data from Data Sources
#
# request <feedset> <pattern> <hostname pattern>
#
# 1/3/2003 Request lightning data from Albany:
# address@hidden (David Knight)
#
request NLDN ".*" striker.atmos.albany.edu PRIMARY
#
# Request data from FSL - ACARS UATurbulence data
# Request data from FSL - AMDAR data
#
request PCWS "^FSL\.NetCDF\.ACARS\.QC\." eldm.fsl.noaa.gov
request FSL3 "^FILTER\.ACARS\.UATurbulence\." eldm.fsl.noaa.gov
#
# DIFAX - DIFAX-like products from University of Wisconsin/AOS
# FSL2 - FSL wind profiler
# UNIWISC - GOES satellite image sectors and products from UW/SSEC
request DIFAX|FSL2|UNIWISC ".*" thelma.ucar.edu PRIMARY
request DIFAX|FSL2|UNIWISC ".*" atm.geo.nsf.gov ALTERNATE
#
# IDS|DDPLUS - NOAAPORT global observations
request WMO ".*" thelma.ucar.edu PRIMARY
request WMO ".*" atm.geo.nsf.gov ALTERNATE
#
# NNEXRAD - NOAAPORT NEXRAD Level III data
request NNEXRAD ".*" thelma.ucar.edu PRIMARY
request NNEXRAD ".*" atm.geo.nsf.gov ALTERNATE
#
# CONDUIT - High resolution model output from NCEP
request CONDUIT ".*" thelma.ucar.edu PRIMARY
request CONDUIT ".*" f5.aos.wisc.edu ALTERNATE
#
# NEXRD2 - NEXRAD Level-2 data NEXRD2 / NMC3
request NEXRD2 ".*" thelma.ucar.edu PRIMARY
#
# EXP
request EXP ".*" level.rap.ucar.edu
request EXP ".*" turb.rap.ucar.edu
#
# New WSI data feed (10/21/1998 - Celia Chen RAP/NCAR)
request WSI ".*" iita.rap.ucar.edu
#request WSI ".*" wsihcsn.unidata.ucar.edu
###############################################################################
# Begin Access control
###############################################################################
###############################################################################
# ALLOW: Who we are willing to feed
# Allow Entries
# allow <feedset> <hostname pattern>
###############################################################################
#
# Giving permission for a Data Sink to perform a request to your LDM
#
# allow <feedset> <hostname pattern>
#
# Giving permission to your own machine and Unidata
#
# Under no circumstances comment out the next allow entry to localhost
# The LDM will NOT start if the lines are commented out.
#
allow ANY
^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$
))
allow ANY
^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.rap\.ucar\.edu\.?$))
###############################################################################
# send anything to your own machine (and other RAP LDM servers)
allow ANY chisel
allow ANY level
allow ANY iita
allow ANY ofour
allow ANY front
allow ANY turb
allow ANY moonbow.rap.ucar.edu
allow ANY serac.rap.ucar.edu
allow ANY saint.rap.ucar.edu
allow ANY
^((localhost|loopback)|(127\.0\.0\.1\.?$)|(^[a-z].*\.rap\.ucar\.edu\.?$))
# ATEC machines (128.117.200.84 atec-ingest)
# ATEC machines (128.117.200.11 mm5-dev)
# ATEC machines (128.117.200.101 uae-c1)
allow ANY uae-c1.rap.ucar.edu
allow ANY atec-ingest
allow ANY mm5-dev
#
# Ceiling and Visibilities Project:
allow ANY ceil
allow ANY vis
allow ANY shaggy
allow ANY scrappy
allow ANY scooter
allow ANY speeder
allow ANY wan
allow ANY scooby
# AWC machines
allow ANY-WSI-NLDN adds.aviationweather.gov
allow ANY-WSI-NLDN amelia.kc.noaa.gov
allow ANY-WSI-NLDN orville.kc.noaa.gov
allow ANY-WSI-NLDN wilbur.kc.noaa.gov
allow ANY-WSI-NLDN 204.227.127.34
allow ANY-WSI-NLDN 204.227.127.38
allow ANY-WSI-NLDN 204.227.127.39
allow ANY-WSI-NLDN 204.227.127.40
allow EXP adds.awc-kc.noaa.gov
allow EXP amelia.kc.noaa.gov
allow EXP orville.kc.noaa.gov
allow EXP wilbur.kc.noaa.gov
#
# UCAR LDM servers
allow ANY thelma.ucar.edu
allow ANY ldm.comet.ucar.edu
allow ANY gizmo.comet.ucar.edu
allow ANY zero.unidata.ucar.edu
allow ANY box.mmm.ucar.edu
allow ANY tea.mmm.ucar.edu
allow ANY 128.117.88.210
#
# FSL LDM servers
allow ANY-WSI-NLDN eldm.fsl.noaa.gov
allow ANY-WSI-NLDN 137.75.1.51
# WSI LDM servers
allow ANY-WSI-NLDN wsi-204-115.wsi.com
# Other outside Users
allow ANY-WSI atm.geo.nsf.gov
allow EXP sirocco.srcc.lsu.edu
allow EXP rainbow.eas.purdue.edu
allow EXP sunset.meteor.wisc.edu
allow EXP unidata.ssec.wisc.edu
allow EXP grayskies.atmos.washington.edu
allow EXP drier.atmos.washington.edu
allow EXP dry.atmos.washington.edu
allow EXP redwood.atmos.albany.edu
allow WMO wnnh001.wind.ttu.edu
#
###############################################################################
# Accept Entries
###############################################################################
# ACCEPT: Who can feed us, currently this action is ONLY needed for WSI data
#
# accept <feedset> <pattern> <hostname pattern>
#
# accept anything from yourself
#
#accept ANY ".*" ^((localhost|loopback)|(127\.0\.0\.1\.?$))
#
# accept from your upstream site
#
# WSI is using ldm4 protocol so the accept is still required
#accept WSI
# .*
# ^[a-z].*\.uni\.wsicorp\.com$
#
###############################################################################
# End
###############################################################################
If you have any questions on the above changes, or if you would like
us to tune up your ldmd.conf file please let me know.
Tom
--- chisel.rap.ucar.edu's ldmd.conf file ---
>#####
># $Id: ldmd.conf,v 1.16 2002/06/17 21:12:43 anne Exp $
># Sample ldmd.conf for ldm5
>####
>#
># This is the main configuration file for the LDM server. All lines that start
># with a "#" sign are comments.
>#
># To debug an LDM that hangs on start up, run the following from LDM home:
># % bin/rpc.ldmd -vl - -q data/ldm.pq etc/ldmd.conf
>#
># If the LDM still hangs, comment out all lines in this file except noted
># allow line entry below, try again.
>#
>##############################################################################
> #
># Exec Entries
>##############################################################################
> #
>#
># Programs that can share a queue with rpc.ldmd server, these programs
># are started by the "exec" command and are in the same process group.
>#
># Regarding pqbinstats, while pqbinstats will generate .stats files in
># your logs directory, it is necessary to also run 'ldmadmin dostats'
># once per hour via cron in order to email the stats to the UPC and
># remove the .stats files that are no longer needed. See
># http://www.unidata.ucar.edu/packages/ldm/ldmConfigInstallList.html#s7
># for more information.
>#exec "pqbinstats"
>exec "pqbinstats -d /home/ldm/logs"
>exec "pqact -l /home/ldm/logs/pqact.log -d /home/ldm /home/ldm/etc/pqact.con
> f"
>exec "pqsurf -p ^(SACN43|S[APX]|FXUS43) -Q /home/ldm/data/pqsurf.pq -l /home
> /ldm/logs/pqsurf.log -d /home/ldm /home/ldm/etc/pqsurf.conf"
>#
>#
># rtstats uses the LDM to send product latency statistics to the UPC.
># We are currently evolving from pqbinstats to rtstats, but in the
># mean time we hope sites will run both. For more info on rtstats
># please see the man page.
># exe "rtstats -q /home/ldm/data/ldm.pq -d /home/ldm/logs -h rtstats.unidata.
> ucar.edu"
>exec "rtstats -h rtstats.unidata.ucar.edu"
># send info back to local disk - NOT working:
>#exec "rtstats -q /home/ldm/data/ldm.pq -d /home/ldm/logs -h "
>#
>##############################################################################
> #
># Request Entries
>##############################################################################
> #
>#
># LDM5 servers request data from Data Sources
>#
># request <feedset> <pattern> <hostname pattern>
>#
>#request WMO ".*" uni0.unidata.ucar.edu
>#
># 1/3/2003 Request lightning data from Albany:
># address@hidden (David Knight)
>#
>request NLDN ".*" striker.atmos.albany.edu PRIMARY
>request NLDN ".*" atm.geo.nsf.gov ALTERNATE
>#
># Request data from FSL - ACARS UATurbulence data
># Request data from FSL - AMDAR data
>#
>request PCWS
> "^FSL\.NetCDF\.ACARS\.QC\."
> eldm.fsl.noaa.gov
>
>request FSL3
> "^FILTER\.ACARS\.UATurbulence\."
> eldm.fsl.noaa.gov
>#
># Requesting data from thelma
># FSL2 - Profiler data
>
>request FSL2 ".*" thelma.ucar.edu PRIMARY
>request FSL2 ".*" atm.geo.nsf.gov ALTERNATE
>
>#request IDS|DDPLUS
>request WMO ".*" thelma.ucar.edu PRIMARY
>request WMO ".*" atm.geo.nsf.gov ALTERNATE
>
>request EXP
> ".*"
> level.rap.ucar.edu
>request EXP
> ".*"
> turb.rap.ucar.edu
>
># Greg Thompson's (1/2003 Celia Chen)
>#request EXP
># ".*"
># 129.118.105.15
>
># DIFAX feed
>request DIFAX ".*" thelma.ucar.edu PRIMARY
>request DIFAX ".*" atm.geo.nsf.gov ALTERNATE
>
>#
># get NEXRAD data
>
>request NEXRAD ".*" thelma.ucar.edu PRIMARY
>request NEXRAD ".*" atm.geo.nsf.gov ALTERNATE
>
>request NMC2 ".*" thelma.ucar.edu PRIMARY
>request NMC2 ".*" atm.geo.nsf.gov ALTERNATE
>
># Level-2 data: NEXRD2 / NMC3
>
>request NEXRD2 ".*" thelma.ucar.edu PRIMARY
>request NEXRD2 ".*" atm.geo.nsf.gov ALTERNATE
>
>#
>request MCIDAS ".*" thelma.ucar.edu PRIMARY
>request MCIDAS ".*" atm.geo.nsf.gov ALTERNATE
>
># New WSI data feed (10/21/1998 - Celia Chen RAP/NCAR)
>
>request WSI ".*" iita.rap.ucar.edu
>#request WSI ".*" wsihcsn.unidata.ucar.edu
>##############################################################################
> #
># Begin Access control
>##############################################################################
> #
>
>##############################################################################
> #
># ALLOW: Who we are willing to feed
># Allow Entries
># allow <feedset> <hostname pattern>
>##############################################################################
> #
>#
># Giving permission for a Data Sink to perform a request to your LDM
>#
># allow <feedset> <hostname pattern>
>#
># Giving permission to your own machine and Unidata
>#
># Under no circumstances comment out the next allow entry to localhost
># The LDM will NOT start if the lines are commented out.
>#
>allow ANY
> ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$
> ))
>allow ANY
> ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.rap\.ucar\.edu\.?$))
>##############################################################################
> #
>
># send anything to your own machine (and other RAP LDM servers)
>
>allow ANY chisel
>allow ANY level
>allow ANY iita
>allow ANY ofour
>allow ANY front
>allow ANY turb
>
>allow ANY moonbow.rap.ucar.edu
>allow ANY serac.rap.ucar.edu
>allow ANY saint.rap.ucar.edu
>allow ANY ^((localhost|loopback)|(127\.0\.0\.1\.?$)|(^[a-z].*\.rap\.ucar
> \.edu\.?$))
>
># ATEC machines (128.117.200.84 atec-ingest)
># ATEC machines (128.117.200.11 mm5-dev)
># ATEC machines (128.117.200.101 uae-c1)
>allow ANY uae-c1.rap.ucar.edu
>allow ANY atec-ingest
>allow ANY mm5-dev
>#
># Ceiling and Visibilities Project:
>allow ANY ceil
>allow ANY vis
>allow ANY shaggy
>allow ANY scrappy
>allow ANY scooter
>allow ANY speeder
>allow ANY wan
>allow ANY scooby
>
># AWC machines
>allow ANY adds.aviationweather.gov
>allow ANY amelia.kc.noaa.gov
>allow ANY orville.kc.noaa.gov
>allow ANY wilbur.kc.noaa.gov
>allow ANY 204.227.127.34
>allow ANY 204.227.127.38
>allow ANY 204.227.127.39
>allow ANY 204.227.127.40
>allow EXP adds.awc-kc.noaa.gov
>allow EXP amelia.kc.noaa.gov
>allow EXP orville.kc.noaa.gov
>allow EXP wilbur.kc.noaa.gov
>#
># UCAR LDM servers
>allow ANY thelma.ucar.edu
>allow ANY ldm.comet.ucar.edu
>allow ANY gizmo.comet.ucar.edu
>allow ANY zero.unidata.ucar.edu
>allow ANY box.mmm.ucar.edu
>allow ANY tea.mmm.ucar.edu
>allow ANY 128.117.88.210
>#
># FSL LDM servers
>allow ANY eldm.fsl.noaa.gov
>allow ANY 137.75.1.51
># WSI LDM servers
>allow ANY wsi-204-115.wsi.com
>
># Other outside Users
>allow ANY-WSI atm.geo.nsf.gov
>allow EXP sirocco.srcc.lsu.edu
>allow EXP rainbow.eas.purdue.edu
>allow EXP sunset.meteor.wisc.edu
>allow EXP unidata.ssec.wisc.edu
>allow EXP grayskies.atmos.washington.edu
>allow EXP drier.atmos.washington.edu
>allow EXP dry.atmos.washington.edu
>allow EXP redwood.atmos.albany.edu
>allow WMO wnnh001.wind.ttu.edu
>#
>##############################################################################
> #
># Accept Entries
>##############################################################################
> #
># ACCEPT: Who can feed us, currently this action is ONLY needed for WSI data
>#
># accept <feedset> <pattern> <hostname pattern>
>#
># accept anything from yourself
>#
>#accept ANY ".*" ^((localhost|loopback)|(127\.0\.0\.1\.?$))
>#
># accept from your upstream site
>#
># WSI is using ldm4 protocol so the accept is still required
>#accept WSI
># .*
># ^[a-z].*\.uni\.wsicorp\.com$
>#
>##############################################################################
> #
># End
>##############################################################################
> #
>From address@hidden Tue Jul 15 14:05:10 2003
Tom,
I want to thank you so very much for cleaning up my ldmd.conf file, which I
adapted many years ago and has some pretty old stuff in it. - like my old
garage at home. ;)
I also appreciate your suggestion in combining some request lines. I have
noticed that there are a lot of rpc.ldmd lines when I do "ldmadmin ps". I
just assumed that it was normal. Silly me!
Thank you also for suggesting to request the CONDUIT stream from
the University of Wisconsin/AOS machine. I will send Pete an email
asking him to add an allow for chisel. I guess I didn't understand
fully Jeff's message when he mentioned something about these special
data feeds.
Many thanks again.
Celia
