[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #PGJ-802823]: request for approval of LDM data receiving



Hi Bei,

re:
> This morning I use the ldm watch,

OK, I assume you mean 'ldmadmin watch'...

re:
> On 11:15, data flow is shown, and after I restart the ldm, WAIT FOR 45
> MINUTES, on 12:00, the data flow is shown again. So, I guess when I
> can't see the data flow, it's because RO event is over?

Yes, I believe so.  The data does not flow continuously, it is episodic.
It is likely that you have been unlucky enough to hit the end of
a transmission sequence.

re:
> When I can't see data flow by ldm watch, I tried notifyme -vl- -f ANY,
> the output is:
> [ldm@Bei ~]$ notifyme -vl- -f ANY
> Feb 28 19:26:59 notifyme[3352] NOTE: Starting Up: localhost:
> 20110228192659.250 TS_ENDT {{ANY,  ".*"}}
> Feb 28 19:26:59 notifyme[3352] NOTE: LDM-5 desired product-class:
> 20110228192659.250 TS_ENDT {{ANY,  ".*"}}
> Feb 28 19:26:59 notifyme[3352] INFO: Resolving localhost to 127.0.0.1
> took 0.009469 seconds
> Feb 28 19:26:59 notifyme[3352] NOTE: NOTIFYME(localhost): OK

OK, this shows two things:

- 'notifyme' was able to contact the LDM top level process on the
  local machine (this is a GOOD thing :-)

- there was no data being received at the time you ran 'notifyme'

Comment:

- 'notifyme' is an incredible useful utility.  It can not only show
  what you are receiving "now"; it can also show what you received
  for some period in the past (the length of time you can see
  "into to past" depends on how large your LDM queue is.  For instance,
  to see all of the products received by your LDM in the past hour\
  (i.e., in the past 3600 seconds), you would run:

<as 'ldm'>
notifyme -vl- -f ANY -o 3600

- 'notifyme' can be used to see what the upstream host you are trying
  to ingest data from is receiving now, or for 'n' seconds in the
  past.  For instance:

  Show what products the upstream host idd.unidata.ucar.edu is receiving
  in the EXP datastream:

<as 'ldm'>
notifyme -vl- -f EXP -h idd.unidata.ucar.edu

  Show what Cosmic products the upstream idd.unidata.ucar.edu has received
  in the EXP feed type in the past hour (3600 seconds):

<as 'ldm'>
notifyme -vl- -f EXP -h idd.unidata.ucar.edu -o 3600

- 'notifyme' accepts a "pattern" through the '-p' flag.  This allows you
  to narrow the list of products you want reporting on.  For instance,

<as 'ldm'>
notifyme -vl- -f EXP -h idd.unidata.ucar.edu -o 3600 -p cosmicrt

  tells 'notifyme' to report on the products in the EXP feedtype whose
  product IDs match the regular experssion "cosmicrt" (i.e., that have
  'cosmicrt' in the product IDs).  The pattern for the '-p' flag
  can be specified to be very simple (like the example just stated) or
  much more complex.  It is the latter capability that is of use to
  the end-user who wants to figure out how to request a specific subset
  of products in a data feed, and/or to fashion an extended regular
  expression for a pattern-action file action.

So, I recommend using 'notifyme' to learn what the products look like
that have been received in a particular datastream:

<as 'ldm'>
notifyme -vl- -f EXP -p cosmicrt -o 3600

Feb 28 21:46:24 notifyme[15425] NOTE: Starting Up: localhost: 
20110228204624.941 TS_ENDT {{EXP,  "cosmicrt"}}
Feb 28 21:46:24 notifyme[15425] NOTE: LDM-5 desired product-class: 
20110228204624.941 TS_ENDT {{EXP,  "cosmicrt"}}
Feb 28 21:46:24 notifyme[15425] INFO: Resolving localhost to 127.0.0.1 took 
0.00068 seconds
Feb 28 21:46:24 notifyme[15425] NOTE: NOTIFYME(localhost): OK
Feb 28 21:46:25 notifyme[15425] INFO:   447096 20110228211011.551     EXP 000  
/pub_out/cosmicrt/level2/atmPrf/2011.059/atmPrf_C001.2011.059.19.08.G15_0001.0001_nc
Feb 28 21:46:25 notifyme[15425] INFO:   448716 20110228211011.589     EXP 000  
/pub_out/cosmicrt/level2/atmPrf/2011.059/atmPrf_C001.2011.059.19.08.G28_0001.0001_nc
Feb 28 21:46:25 notifyme[15425] INFO:   498096 20110228211011.639     EXP 000  
/pub_out/cosmicrt/level2/atmPrf/2011.059/atmPrf_C001.2011.059.19.27.G22_0001.0001_nc
Feb 28 21:46:25 notifyme[15425] INFO:   544076 20110228211011.753     EXP 000  
/pub_out/cosmicrt/level2/atmPrf/2011.059/atmPrf_C001.2011.059.19.58.G28_0001.0001_nc
Feb 28 21:46:26 notifyme[15425] INFO:   415096 20110228211011.818     EXP 000  
/pub_out/cosmicrt/level2/atmPrf/2011.059/atmPrf_C001.2011.059.19.58.G31_0001.0001_nc
Feb 28 21:46:26 notifyme[15425] INFO:   466996 20110228211011.838     EXP 000  
/pub_out/cosmicrt/level2/atmPrf/2011.059/atmPrf_C001.2011.059.20.09.G11_0001.0001_nc
 ...

We can see from this (abbreviated) listing that Cosmic product IDs look like:

/pub_out/cosmicrt/level2/atmPrf/2011.059/atmPrf_C005.2011.059.17.22.G10_0001.0001_nc
/pub_out/cosmicrt/level2/atmPrf/2011.059/atmPrf_C005.2011.059.17.32.G16_0001.0001_nc
 ...

Using this information, we can fashion an extended regular expression to test:

notifyme -vl- -f EXP -o 1000000 -p 
'([12][0-9][0-9][0-9])\.([0-3][0-9][0-9])/(bfr|ion|wet)Prf(.*_nc)'

Feb 28 21:48:36 notifyme[15635] NOTE: Starting Up: localhost: 
20110217080156.706 TS_ENDT {{EXP,  
"([12][0-9][0-9][0-9])\.([0-3][0-9][0-9])/(bfr|ion|wet)Prf(.*_nc)"}}
Feb 28 21:48:36 notifyme[15635] NOTE: LDM-5 desired product-class: 
20110217080156.706 TS_ENDT {{EXP,  
"([12][0-9][0-9][0-9])\.([0-3][0-9][0-9])/(bfr|ion|wet)Prf(.*_nc)"}}
Feb 28 21:48:36 notifyme[15635] INFO: Resolving localhost to 127.0.0.1 took 
0.000678 seconds
Feb 28 21:48:36 notifyme[15635] NOTE: NOTIFYME(localhost): OK
Feb 28 21:48:37 notifyme[15635] INFO:    11264 20110228212018.886     EXP 000  
/pub_out/cosmicrt/level2/ionPrf/2011.059/ionPrf_C005.2011.059.17.32.G17_0001.0001_nc
Feb 28 21:48:37 notifyme[15635] INFO:    11912 20110228212018.910     EXP 000  
/pub_out/cosmicrt/level2/ionPrf/2011.059/ionPrf_C005.2011.059.17.35.G04_0001.0001_nc
Feb 28 21:48:37 notifyme[15635] INFO:    13040 20110228212018.919     EXP 000  
/pub_out/cosmicrt/level2/ionPrf/2011.059/ionPrf_C005.2011.059.17.49.G20_0001.0001_nc
Feb 28 21:48:37 notifyme[15635] INFO:    11624 20110228212019.032     EXP 000  
/pub_out/cosmicrt/level2/ionPrf/2011.059/ionPrf_C005.2011.059.17.57.G31_0001.0001_nc
Feb 28 21:48:37 notifyme[15635] INFO:    13160 20110228212019.095     EXP 000  
/pub_out/cosmicrt/level2/ionPrf/2011.059/ionPrf_C005.2011.059.17.57.G32_0001.0001_nc
Feb 28 21:48:37 notifyme[15635] INFO:    11312 20110228212019.112     EXP 000  
/pub_out/cosmicrt/level2/ionPrf/2011.059/ionPrf_C005.2011.059.18.12.G16_0001.0001_nc
Feb 28 21:48:37 notifyme[15635] INFO:    11696 20110228212019.139     EXP 000  
/pub_out/cosmicrt/level2/ionPrf/2011.059/ionPrf_C005.2011.059.18.23.G03_0001.0001_nc
 ...

OK, now we have a useful extended regular expression that can be useful in
a pattern-action file action:

([12][0-9][0-9][0-9])\.([0-3][0-9][0-9])/(bfr|ion|wet)Prf(.*_nc)

So, a pattern-action file action that should work to capture Cosmic
data in the EXP feed type would be:

EXP   ([12][0-9][0-9][0-9])\.([0-3][0-9][0-9])/(bfr|ion|wet)Prf(.*_nc)
    FILE    -close
    data/Cosmic/\1\2/\3Prf/\3Prf\4

NOTE: make sure that all whitespace in this pqact.conf entry are tabs,
not spaces!

This should create files in the ~ldm/data/Cosmic/2011059/ionPrf/ directory,
and the file names should look like:

ionPrf_C005.2011.059.18.23.G03_0001.0001_nc

re:
> I guess it's normal, and I think this problem is shot....

I am not sure I understand this, but no worries...

re:
> So till now the only problem is data can not be saved and logging.

The pattern I sent you earlier was incorrect -- it assumed that
there was a '.' between the 'nc' at the end of the Product IDs
when the character right before the 'nc' is an '_' (underscore).

You should use the new pattern-action file entry above in replacement
of the one I sent you earlier.  Again, remember that anytime you
make a modification to a pattern-action file action, you should
check to make sure that there were no obvious/gross/big errors:

<as 'ldm'>
ldmadmin pqactcheck
ldmadmin pqactHUP    <- if no errors were reported from 'ldmadmin pqactcheck'

re:
> I attached all the important conf files to you. And the 'ldm'
> permission for all these conf files and ldmd.log is rw. And also
> hupsyslog is setuid to root:
> 
> -rwsr-xr-x. 1 root ldm 11246 Feb 18 17:12 /home/ldm/bin/hupsyslog

Very good.  I have edited the pqact.conf file and updated the regular
expression pattern in the action designed to FILE Cosmic data.  The
updated pqact.conf file is attached to this reply.

re:
> I don't know if there is anything else I can let you know, but
> do you mind remote logon my computer to check all the configurations?...
> My IP is 136.129.122.134 ...

I can not SSH to your machine.  I suspect that either a firewall is
blocking SSH access from the outside, or the SSH daemon (sshd) is
not running.

Please replace your current ~ldm/etc/pqact.conf file with the copy I attached
to this email and then stop and restart your LDM:

<as 'ldm'>
cd
cd etc
-- replace pqact.conf
cd
ldmadmin stop
ldmadmin start

-- OR --

ldmadmin restart

Please let me know if you continue to have problems saving the Cosmic data to
disk.

Now, on the topic of LDM logging, please try the following:

<as 'ldm'>
cd
ls -alt                               <- send me the output listing

ls -alt /home/ldm/var/logs/ldmd.log   <- send me the output listing

logger -p local0.debug 'test of LDM logging'

ls -alt /home/ldm/var/logs/ldmd.log   <- send me the output listing

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: PGJ-802823
Department: Support IDD
Priority: Normal
Status: Closed

Attachment: pqact.conf
Description: Binary data