[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #LRB-829354]: Issues with product-queue



Hi Tim,

re:
> Okay, this logging seems to be the case.

I interpret this to mean that logging is not working.

re:
> How would I properly set this up?

I logged back into your system, and I saw that that the things
that need to be setup are, in fact, setup for LDM logging to
work.  I then noticed that you are running Ubuntu Linux, and
I understood what is going on.  For some reason in Ubuntu
rsyslogd to write to your LDM log file; this is because the
read/write/execute permissions on ldmd.log are typically setup
as 664 (-rw-rw-r--), so the file not being writable by world
is the problem.

A workaround for this is to set the 'umask' for 'ldm' to be
'000' (it is currently 002, which is normal).  I just implemented
this by adding:

umask 000

to the top of ~ldm/.bashrc.  I tested the change by logging off
and then logging back on to your machine.  After doing this, I
deleted and remade the LDM log file:

<as 'ldm'>

rm logs/ldmd.log && touch logs/ldmd.log

and then restarted the LDM:

ldmadmin restart

After doing this LDM logging is working; this can be see by the
non-zero size of the LDM log file:

:~$ ls -alt ~ldm/logs/ldmd.log
-rw-rw-rw- 1 ldm ldm 7568 Aug 21 15:41 /home/ldm/logs/ldmd.log

The next thing I did was to run 'notifyme' to see if:

1) you are ALLOWed to REQUEST NEXRAD2 data from the upstream hosts
   you want to REQUEST from

   You are ALLOWed.

2) if you were getting any data

   You are not getting any data.

The next thing I did was to examine the ~ldm/etc/ldmd.conf REQUEST
lines you have for NEXRAD2 data:

:~$ grep -i ^request etc/ldmd.conf
REQUEST NEXRAD2 "/p...(UDX|ABR|FSD)" idd.tamu.edu
REQUEST NEXRAD2 "/p...(UDX|ABR|FSD)" idd.meteo.psu.edu
REQUEST NEXRAD2 "/p...(UDX|ABR|FSD)" idd.unidata.ucar.edu

If you look at the LDM/IDD Product IDs for the NEXRAD2 data:

:~$ notifyme -vl- -f NEXRAD2 -h idd.tamu.edu -o 600
Aug 21 21:44:33 notifyme[9273] NOTE: Starting Up: idd.tamu.edu: 
20140821213433.705 TS_ENDT {{NEXRAD2,  ".*"}}
Aug 21 21:44:33 notifyme[9273] NOTE: LDM-5 desired product-class: 
20140821213433.705 TS_ENDT {{NEXRAD2,  ".*"}}
Aug 21 21:44:33 notifyme[9273] INFO: Resolving idd.tamu.edu to 128.194.76.175 
took 0.002042 seconds
Aug 21 21:44:33 notifyme[9273] NOTE: NOTIFYME(idd.tamu.edu): OK
Aug 21 21:44:34 notifyme[9273] INFO:    40786 20140821213433.740 NEXRAD2 502065 
 L2-BZIP2/KILX/20140821213042/502/65/I/V06/0
Aug 21 21:44:34 notifyme[9273] INFO:    47088 20140821213433.733 NEXRAD2 40066  
L2-BZIP2/KDGX/20140821213043/40/66/I/V06/0
Aug 21 21:44:34 notifyme[9273] INFO:    65527 20140821213433.736 NEXRAD2 222050 
 L2-BZIP2/KUDX/20140821213138/222/50/I/V06/0
Aug 21 21:44:35 notifyme[9273] INFO:    37829 20140821213433.809 NEXRAD2 88006  
L2-BZIP2/KDOX/20140821213400/88/6/I/V07/0
Aug 21 21:44:35 notifyme[9273] INFO:    51051 20140821213433.973 NEXRAD2 941017 
 L2-BZIP2/KGJX/20140821213349/941/17/I/V06/0
Aug 21 21:44:35 notifyme[9273] INFO:    76064 20140821213433.795 NEXRAD2 541057 
 L2-BZIP2/KCYS/20140821213133/541/57/I/V06/0
 ...

You will see that the pattern you are using in your REQUEST lines will never
be matched since there is no '/p' in any NEXRAD2 Product ID.

I think that what you are trying to REQUEST is the set of NEXRAD Level 2
stations that have (UDX|ABR|FSD) as part of their name.  Given this
assumption, I went ahead and redid your ldmd.conf REQUEST lines and then
restarted your LDM:

#request Nexrad level 2 data

#REQUEST NEXRAD2 "/p...(UDX|ABR|FSD)" idd.tamu.edu
#REQUEST NEXRAD2 "/p...(UDX|ABR|FSD)" idd.meteo.psu.edu
#REQUEST NEXRAD2 "/p...(UDX|ABR|FSD)" idd.unidata.ucar.edu

REQUEST NEXRAD2 "(UDX|ABR|FSD)" idd.tamu.edu
REQUEST NEXRAD2 "(UDX|ABR|FSD)" idd.meteo.psu.edu
REQUEST NEXRAD2 "(UDX|ABR|FSD)" idd.unidata.ucar.edu

After editing ldmd.conf and restarting, LOTs of Level 2 data is
being received:

:~$ ldmadmin watch
(Type ^D when finished)
Aug 21 21:50:43 pqutil INFO:   127516 20140821213134.410 NEXRAD2 419038  
L2-BZIP2/KABR/20140821212927/419/38/I/V06/0
Aug 21 21:50:43 pqutil INFO:    59277 20140821213136.329 NEXRAD2 887021  
L2-BZIP2/KFSD/20140821213040/887/21/I/V06/0
Aug 21 21:50:43 pqutil INFO:    58197 20140821213138.364 NEXRAD2 887022  
L2-BZIP2/KFSD/20140821213040/887/22/I/V06/0
Aug 21 21:50:43 pqutil INFO:    10142 20140821213139.391 NEXRAD2 222001  
L2-BZIP2/KUDX/20140821213138/222/1/S/V06/0
Aug 21 21:50:43 pqutil INFO:    60988 20140821213140.401 NEXRAD2 887023  
L2-BZIP2/KFSD/20140821213040/887/23/I/V06/0
Aug 21 21:50:43 pqutil INFO:   129924 20140821213139.486 NEXRAD2 419039  
L2-BZIP2/KABR/20140821212927/419/39/I/V06/0
Aug 21 21:50:43 pqutil INFO:   141280 20140821213142.474 NEXRAD2 222002  
L2-BZIP2/KUDX/20140821213138/222/2/I/V06/0
Aug 21 21:50:43 pqutil INFO:    54057 20140821213143.439 NEXRAD2 887024  
L2-BZIP2/KFSD/20140821213040/887/24/I/V06/0
Aug 21 21:50:43 pqutil INFO:   135551 20140821213143.561 NEXRAD2 419040  
L2-BZIP2/KABR/20140821212927/419/40/I/V06/0
Aug 21 21:50:43 pqutil INFO:    62913 20140821213145.483 NEXRAD2 887025  
L2-BZIP2/KFSD/20140821213040/887/25/I/V06/0
Aug 21 21:50:43 pqutil INFO:   101175 20140821205101.110 NEXRAD2 876028  
L2-BZIP2/KFSD/20140821204946/876/28/I/V06/0
Aug 21 21:50:43 pqutil INFO:   194473 20140821213145.571 NEXRAD2 222003  
L2-BZIP2/KUDX/20140821213138/222/3/I/V06/0
 ...

The final thing was to see if any products were being written to disk; they
were not.  I then took a look at your pattern-action file action to see
what you are trying to do.  The only uncommented action I found was:

NEXRAD  ^SDUS5. .... ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...) FILE    
-close  data/nexrad/\5/\4/\4_(\1:yyyy)(\1:mm)\1_\2\3

This will not match anything, since the feed type you are specifying to
check, 'NEXRAD', is not what you are REQUESTing.  The feed type for
NEXRAD Level 2 data is 'NEXRAD2', not NEXRAD which an equivalent name
for NEXRAD3 which is the feed of NEXRAD Level III products.

So, the question is which NEXRAD data do you want/need, Level II or
Level III?

If you want/need Level II data (full volume scan data), we need to change
the action entry in your ~ldm/etc/pqact.conf file.

If you want/need Level III data (NEXRAD products), we need to change the
REQUEST lines in your ~ldm/etc/ldmd.conf file.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: LRB-829354
Department: Support IDD
Priority: Normal
Status: Closed