[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[GEMPAK #VFW-525515]: A bit more about why the decoders might not be working....

Raymond,

I made a few edits to ldmd.conf and the pqact files pqact.conf and pqact.gempak.

I think the root of the problem was the relative path name "decoders/dcgrib2" 
in the pattern actions for decoders.  I removed "decoders/" from all entries, 
and also changed GEMTBL in these files to point to "/home/gempak/NAWIPS/..." 
rather than "/home/gempak/GEMAK6.7.0/...", which isn't responsible for the 
problem, but is good practice for when upgrading to a new version.

Lastly, I changed the EXEC lines in ldmd.conf to specific the feed type and 
pqact file using the fully-qualified pathname:

EXEC    "pqact -f UNIDATA /usr/local/ldm/etc/pqact.conf"
EXEC    "pqact -f NGRID|CONDUIT|HDS /usr/local/ldm/etc/pqact.gempak"

After this, I noticed model data decoded to ~ldm/data/data/gempak - still the 
double "data" in the path.

I redefined LDMHOME in user gempak's Gemenviron.profile and re-sourced the file 
into the gempak user's environment, then restarted the ldm again from the ldm 
account.

I have yet to see model data written to ~ldm/data/gempak/model/ but I'm going 
to keep it running for an hour or so and come back to look at it later.  Feel 
free to check it out and make adjustments as needed.  I have to go to a meeting 
and will check back in this afternoon.

Michael


> Michael,
> 
> I'm still stuck on this write problem with the GEMPAK decoders.  I
> have checked and double-checked the env variables and they seem fine
> and the permissions to write to the data/gempak/surface and
> data/gempak/log directory is 777 so anyone can write to it.
> 
> Is there a way that I can invoke this decoder:
> 
> WMO   ^S[AP]
> PIPE  decoders/dcmetr -v 2 -a 500 -m 72 -s sfmetar_sa.tbl
> -d data/gempak/logs/dcmetr.log
> -e GEMTBL=/home/gempak/GEMPAK6.7.0/gempak/tables
> data/gempak/surface/YYYYMMDD_sao.gem
> 
> on a file rather than going through the PIPE to see specifically what
> is happening here.  Can I run decoders/dcmetr on a file to check this
> out?  Do you have a simple, few line test file that dcmetr should
> decode to /surface/YYYMMDD_sao.gem?
> 
> Ray
> 
> On Thu, Dec 6, 2012 at 1:20 PM, Unidata GEMPAK Support
> <address@hidden> wrote:
> > I'm going to suggest the sym link $OS_BIN to ~ldm/decoders be removes, and 
> > copy (as user ldm) all dc programs to the directory ~ldm/decoders/ "cp 
> > $OS_BIN/dc* ~ldm/decoders/.
> >
> > Then rehash or re-login and check with "which dcgrib2" and test your 
> > scripts again.  Might simply be a permission issue with user gempak owning 
> > the programs.
> >
> > And like you said, I'm also worried about the double ldm/data/data/ - send 
> > me back the current list of "env" from both gempak and ldm users, there's a 
> > conflict somewhere and I'm still not sure where.
> >
> > -Michael
> >
> >> Michael,
> >>
> >> If I cd to ~ldm/decoders and run a script it seems to start
> >>
> >> If I run the script as decoders/acprof,  I get errors....    See  >>>>>
> >>
> >> [ldm@awipserver decoders]$ acprof
> >> [IP -2]
> >> [IP -10] ACPROF
> >> Creating process: gplt for queue 294914
> >> [LV -3]
> >>
> >>
> >> ^C out of there....
> >>
> >> [ldm@awipserver decoders]$ cd ~
> >> [ldm@awipserver ~]$ decoders/acprof
> >> [IP -2]
> >> [IP -10] ACPROF
> >> [GEMPLT -101]
> >> [ACPROF -3]
> >> Error in message send = 22
> >> itype, ichan, nwords,0,294914,3
> >> [ldm@awipserver ~]$
> >>
> >> Maybe it won't run through the symbolic link?
> >>
> >> Ray
> >> On Wed, Dec 5, 2012 at 5:13 PM, Unidata GEMPAK Support
> >> <address@hidden> wrote:
> >> > As user ldm, confirm on login that the decoders are in your path: "which 
> >> > dcgrib2" should point to ~/decoders/dcgrib2 or $OS_BIN/dcgrib2
> >> >
> >> >
> >> >> Michael,
> >> >>
> >> >> Ok still did not get there......  I made a symbolic link to the
> >> >> directory pointed to by gempak $OS_BIN.  ldm tries to run them out of
> >> >> the gempak directory.
> >> >>
> >> >> decoders -> /home/gempak/GEMPAK6.7.0/os/linux64/bin/
> >> >>
> >> >> All the dc files are permission 755  rwxr-xr-x  so ldm should be able
> >> >> to run them.
> >> >>
> >> >> But if you look at the last ldmd.log  (attached) it says the decoders
> >> >> don't run.   Barring moving them directly to the ldm/decoders
> >> >> directory, see anything else?
> >> >>
> >> >> All the decoder calls are preceded by a "No such file or directory"
> >> >> but without going into the scripts for decoders/dcgrib2 for example, I
> >> >> don't know what directory it is looking for.
> >> >>
> >> >> Ray
> >> >>
> >> >>
> >> >> On Wed, Dec 5, 2012 at 3:01 PM, Unidata GEMPAK Support
> >> >> <address@hidden> wrote:
> >> >> > $LDMHOME for the gempak user must be able to read where the ldm is 
> >> >> > filing and decoding data.  This appears to be setup correctly on your 
> >> >> > system with the only change needed the directory ~ldm/decoders.
> >> >> >
> >> >> > However I don't see GEMDATA in 'env' for the gempak user, though from 
> >> >> > $HDS and other data environmental variables, it appears to be set.
> >> >> >
> >> >> > -Michael
> >> >> >
> >> >> >> Ok,
> >> >> >>
> >> >> >> I must have missed that instruction in the cookbook.  I'll make that 
> >> >> >> fix .
> >> >> >>
> >> >> >> The /usr/local/ldm/data   symbolic link then SHOULD (?) be to
> >> >> >> /mnt/sdb/ldm/var/data  ????
> >> >> >>
> >> >> >> Also ldm and gempak are separate users so why should ldm know about
> >> >> >> the gempak environment variables?  Should they be separate or in a
> >> >> >> group?  Could it be that gempak didn't write the decoders because it
> >> >> >> didn't have permissions in the ldm group?
> >> >> >>
> >> >> >> Thanks, I'll try these fixes later today and let you know the 
> >> >> >> outcome .
> >> >> >>
> >> >> >> Ray
> >> >> >>
> >> >> >> On Wed, Dec 5, 2012 at 2:44 PM, Unidata GEMPAK Support
> >> >> >> <address@hidden> wrote:
> >> >> >> > I don't see a directory ~ldm/decoders to contain the dc* programs 
> >> >> >> > from $OS_BIN.  /usr/local/ldm/decoders is in the ldm user's $PATH, 
> >> >> >> > so it's looking there for decoders but not finding them.  This 
> >> >> >> > explains why some raw data were written - these pattern actions 
> >> >> >> > are using a FILE command rather than PIPE to a decoder.
> >> >> >> >
> >> >> >> > The solution is to copy all dc* programs from $OS_BIN to 
> >> >> >> > ~ldm/decoders, or to symlink the ~ldm/decoders directory to 
> >> >> >> > $OS_BIN.
> >> >> >> >
> >> >> >> > Michael
> >> >> >> >
> >> >> >> >> Michael,
> >> >> >> >>
> >> >> >> >> I took all the filters out of pqact.conf just so I could be sure 
> >> >> >> >> that
> >> >> >> >> the only calls I was making was to a GEMPAK needed file.   I 
> >> >> >> >> actually
> >> >> >> >> had one IDS call open yesterday..... bad form I guess, but that 
> >> >> >> >> didn't
> >> >> >> >> stop pqact.gempak from running.
> >> >> >> >>
> >> >> >> >> Here is the LDM environment:
> >> >> >> >>
> >> >> >> >> [ldm@awipserver ~]$ env
> >> >> >> >> MANPATH=/usr/local/ldm:man:/usr/man
> >> >> >> >> HOSTNAME=awipserver.physics.umbc.edu
> >> >> >> >> TERM=xterm-color
> >> >> >> >> SHELL=/bin/bash
> >> >> >> >> HISTSIZE=1000
> >> >> >> >> SSH_CLIENT=130.85.163.143 49658 22
> >> >> >> >> SSH_TTY=/dev/pts/2
> >> >> >> >> USER=ldm
> >> >> >> >> LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
> >> >> >> >> MAIL=/var/spool/mail/ldm
> >> >> >> >> PATH=/usr/local/ldm/decoders:/usr/local/ldm/util:/usr/local/ldm:bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/ldm/bin
> >> >> >> >> INPUTRC=/etc/inputrc
> >> >> >> >> PWD=/usr/local/ldm
> >> >> >> >> LANG=en_US.UTF-8
> >> >> >> >> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
> >> >> >> >> SHLVL=1
> >> >> >> >> HOME=/usr/local/ldm
> >> >> >> >> LOGNAME=ldm
> >> >> >> >> LDMHOME=/usr/local/ldm
> >> >> >> >> CVS_RSH=ssh
> >> >> >> >> SSH_CONNECTION=130.85.163.143 49658 130.85.163.219 22
> >> >> >> >> LESSOPEN=|/usr/bin/lesspipe.sh %s
> >> >> >> >> G_BROKEN_FILENAMES=1
> >> >> >> >> _=/bin/env
> >> >> >> >> [ldm@awipserver ~]$
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> And, no, "does /usr/local/ldm/data point to /mnt/sbd/var/ldm?"   
> >> >> >> >> No,
> >> >> >> >> it is  /usr/local/ldm/data > /usr/local/ldm/var/data ..... It is 
> >> >> >> >> using
> >> >> >> >> the var as a symbolic link and that should be changed so it 
> >> >> >> >> explicitly
> >> >> >> >> points to the right level on /mnt/sdb....
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> So the symbolic link should be
> >> >> >> >>
> >> >> >> >> ln -s /mnt/sdb/ldm/var/data       data
> >> >> >> >>
> >> >> >> >> for the data symbolic link?
> >> >> >> >>
> >> >> >> >> the parallel structure on /mnt/sdb should be?????
> >> >> >> >>
> >> >> >> >> ldm
> >> >> >> >> var
> >> >> >> >> data    logs   queues?
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Here is the directory for /usr/local/ldm:
> >> >> >> >>
> >> >> >> >> lrwxrwxrwx 1 ldm  ldm    11 Nov 30 13:47 bin -> runtime/bin
> >> >> >> >> -rw-rw-r-- 1 ldm  ldm   214 Dec  3 14:22 crontab
> >> >> >> >> lrwxrwxrwx 1 ldm  ldm    23 Nov 30 13:47 data -> 
> >> >> >> >> /usr/local/ldm/var/data
> >> >> >> >> drwxr-xr-x 2 ldm  ldm  4096 Nov 30 13:26 Desktop
> >> >> >> >> drwxrwxr-x 2 ldm  ldm  4096 Dec  5 09:42 etc
> >> >> >> >> lrwxrwxrwx 1 ldm  ldm    15 Nov 30 13:47 include -> 
> >> >> >> >> runtime/include
> >> >> >> >> drwxr-xr-x 7 ldm  ldm  4096 Nov 30 13:46 ldm-6.11.1
> >> >> >> >> lrwxrwxrwx 1 ldm  ldm    11 Nov 30 13:47 lib -> runtime/lib
> >> >> >> >> lrwxrwxrwx 1 ldm  ldm    23 Nov 30 13:47 logs -> 
> >> >> >> >> /usr/local/ldm/var/logs
> >> >> >> >> lrwxrwxrwx 1 ldm  ldm    10 Dec  3 13:33 runtime -> ldm-6.11.1
> >> >> >> >> lrwxrwxrwx 1 ldm  ldm    13 Nov 30 13:47 share -> runtime/share
> >> >> >> >> lrwxrwxrwx 1 ldm  ldm    11 Nov 30 13:47 src -> runtime/src
> >> >> >> >> lrwxrwxrwx 1 root root   12 Dec  3 13:45 var -> /mnt/sdb/var
> >> >> >> >> -rw-rw-r-- 1 ldm  ldm    25 Dec  5 09:37 watch.txt
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Ray
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Wed, Dec 5, 2012 at 2:20 PM, Unidata GEMPAK Support
> >> >> >> >> <address@hidden> wrote:
> >> >> >> >> > Hi Raymond,
> >> >> >> >> >
> >> >> >> >> > Thanks for attaching the logs and environmental variable files.
> >> >> >> >> >
> >> >> >> >> > The first thing noticed in ldmd.log is
> >> >> >> >> >
> >> >> >> >> > Dec 05 14:35:36 awipserver pqact[19304] NOTE: 
> >> >> >> >> > Configuration-file "/usr/local/ldm/etc/pqact.conf" has no 
> >> >> >> >> > entries. You should probably not start this program instead.
> >> >> >> >> >
> >> >> >> >> > Which suggests the pqact.conf isn't being taken correctly.  
> >> >> >> >> > Even so, there are decoder entries in the log file such as:
> >> >> >> >> >
> >> >> >> >> > Dec 05 14:35:36 awipserver pqact[19308] ERROR: [filel.c:1404] 
> >> >> >> >> > Couldn't execute decoder "decoders/dcnldn"
> >> >> >> >> >
> >> >> >> >> > So I'm not entirely sure what the problem is, but I can guess 
> >> >> >> >> > that it's due to difference between data directories in the 
> >> >> >> >> > gempak and ldm users' environment.
> >> >> >> >> >
> >> >> >> >> > What is 'env' os the user ldm, and does /usr/local/ldm/data 
> >> >> >> >> > point to /mnt/sbd/var/ldm?
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > -Michael
> >> >> >> >> >
> >> >> >> >> >> Michael,
> >> >> >> >> >>
> >> >> >> >> >> Ok, I have LDM and GEMPAK running but not much is getting 
> >> >> >> >> >> decoded by
> >> >> >> >> >> pqact.gempak.   I suspect it is due to an incorrect directory
> >> >> >> >> >> structure in the LDM directory.
> >> >> >> >> >>
> >> >> >> >> >> What I've done:
> >> >> >> >> >>
> >> >> >> >> >> 1) I have ldm and gempak as users on 
> >> >> >> >> >> awipsserver.physics.umbc.edu.
> >> >> >> >> >> ldm's home directory is /usr/local/ldm  (as suggested in the 
> >> >> >> >> >> ldm
> >> >> >> >> >> documentation).  gempak's home directory is /home/gempak  
> >> >> >> >> >> (with other
> >> >> >> >> >> users)
> >> >> >> >> >>
> >> >> >> >> >> 2)  If I run pqact.conf that comes with ldm, I can pull feeds 
> >> >> >> >> >> (see
> >> >> >> >> >> watch.txt attached) from idd.meteo.psu.edu and it looks like 
> >> >> >> >> >> the
> >> >> >> >> >> UNIDATA feed (and the UNIWISC) feeds work.
> >> >> >> >> >>
> >> >> >> >> >> 3)  When I built ldm, I did not want the /var directory on my 
> >> >> >> >> >> main
> >> >> >> >> >> hard drive on the server, but I put it on a raid  
> >> >> >> >> >> (/mnt/sdb/var) and I
> >> >> >> >> >> symbolically linked /var and /data in ldm to point to 
> >> >> >> >> >> /mnt/sdb/var and
> >> >> >> >> >> /mnt/sdb/var/data).   That worked ok and the ldm data was 
> >> >> >> >> >> pushed to
> >> >> >> >> >> /mnt/sdb/var/data/surface/US.... for the demo pqact.conf feeds 
> >> >> >> >> >> from
> >> >> >> >> >> IDS.
> >> >> >> >> >>
> >> >> >> >> >> 4) In GEMPAK, all built out ok (except for the /gf bug that I 
> >> >> >> >> >> sent
> >> >> >> >> >> previously) and GEMPAK runs fine.   I attach the env for 
> >> >> >> >> >> gempak.   I
> >> >> >> >> >> think the issue is here.   I left $GEMDATA as it was in 
> >> >> >> >> >> Gemenviron as
> >> >> >> >> >> it was built by the csh routine in GEMPAK.
> >> >> >> >> >>
> >> >> >> >> >> setenv GEMDATA       /data/ldm/gempak
> >> >> >> >> >>
> >> >> >> >> >> That seemed strange to me but it started putting the gempak 
> >> >> >> >> >> data in
> >> >> >> >> >> /usr/local/ldm/data/data/images/    etc....
> >> >> >> >> >>
> >> >> >> >> >> I only got /images   /jason  and  /nwx  built in that 
> >> >> >> >> >> directory .
> >> >> >> >> >> /surface doesn't get decoded.    From the gempak_env, you can 
> >> >> >> >> >> see that
> >> >> >> >> >> it set up the /usr/local/ldm/data/data/gempak structure.
> >> >> >> >> >>
> >> >> >> >> >> I attach a zipped version of the ldmd.log file with the 
> >> >> >> >> >> errors.  It
> >> >> >> >> >> looks like this is not feed related but the decoders are not 
> >> >> >> >> >> all
> >> >> >> >> >> working.
> >> >> >> >> >>
> >> >> >> >> >> 5) To make GEMPAK read from the right files,  I had to change 
> >> >> >> >> >> the
> >> >> >> >> >> above Gemenviron variable to be
> >> >> >> >> >>
> >> >> >> >> >> setenv GEMDATA      /usr/local/ldm/data/data/gempak
> >> >> >> >> >>
> >> >> >> >> >> and then GEMPAK could go and get the data that was available 
> >> >> >> >> >> (images
> >> >> >> >> >> mostly.... the GOES data is coming in ok.)
> >> >> >> >> >>
> >> >> >> >> >> Suggestions on how to fix the Gemenviron file (also attached)?
> >> >> >> >> >>
> >> >> >> >> >> Ray
> >> >> >> >> >>
> >> >> >> >> >> --
> >> >> >> >> >> Raymond M. Hoff
> >> >> >> >> >> Professor of Physics
> >> >> >> >> >> Rm 426, Physics Dept.
> >> >> >> >> >> University of Maryland, Baltimore County
> >> >> >> >> >> 1000 Hilltop Circle
> >> >> >> >> >> Baltimore MD 21250
> >> >> >> >> >> p: 410-455-1943 f:410-455-1042
> >> >> >> >> >> e: address@hidden
> >> >> >> >> >> physics.umbc.edu/~hoff
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> > Ticket Details
> >> >> >> >> > ===================
> >> >> >> >> > Ticket ID: XDH-337278
> >> >> >> >> > Department: Support GEMPAK
> >> >> >> >> > Priority: Normal
> >> >> >> >> > Status: Open
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> Raymond M. Hoff
> >> >> >> >> Professor of Physics
> >> >> >> >> Rm 426, Physics Dept.
> >> >> >> >> University of Maryland, Baltimore County
> >> >> >> >> 1000 Hilltop Circle
> >> >> >> >> Baltimore MD 21250
> >> >> >> >> p: 410-455-1943 f:410-455-1042
> >> >> >> >> e: address@hidden
> >> >> >> >> physics.umbc.edu/~hoff
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> > Ticket Details
> >> >> >> > ===================
> >> >> >> > Ticket ID: XDH-337278
> >> >> >> > Department: Support GEMPAK
> >> >> >> > Priority: Normal
> >> >> >> > Status: Open
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Raymond M. Hoff
> >> >> >> Professor of Physics
> >> >> >> Rm 426, Physics Dept.
> >> >> >> University of Maryland, Baltimore County
> >> >> >> 1000 Hilltop Circle
> >> >> >> Baltimore MD 21250
> >> >> >> p: 410-455-1943 f:410-455-1042
> >> >> >> e: address@hidden
> >> >> >> physics.umbc.edu/~hoff
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> > Ticket Details
> >> >> > ===================
> >> >> > Ticket ID: XDH-337278
> >> >> > Department: Support GEMPAK
> >> >> > Priority: Normal
> >> >> > Status: Open
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Raymond M. Hoff
> >> >> Professor of Physics
> >> >> Rm 426, Physics Dept.
> >> >> University of Maryland, Baltimore County
> >> >> 1000 Hilltop Circle
> >> >> Baltimore MD 21250
> >> >> p: 410-455-1943 f:410-455-1042
> >> >> e: address@hidden
> >> >> physics.umbc.edu/~hoff
> >> >>
> >> >>
> >> >
> >> >
> >> > Ticket Details
> >> > ===================
> >> > Ticket ID: XDH-337278
> >> > Department: Support GEMPAK
> >> > Priority: Normal
> >> > Status: Open
> >> >
> >>
> >>
> >>
> >> --
> >> Raymond M. Hoff
> >> Professor of Physics
> >> Rm 426, Physics Dept.
> >> University of Maryland, Baltimore County
> >> 1000 Hilltop Circle
> >> Baltimore MD 21250
> >> p: 410-455-1943 f:410-455-1042
> >> e: address@hidden
> >> physics.umbc.edu/~hoff
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: VFW-525515
> > Department: Support GEMPAK
> > Priority: Normal
> > Status: Open
> >
> 
> 
> 
> --
> Raymond M. Hoff
> Professor of Physics
> Rm 426, Physics Dept.
> University of Maryland, Baltimore County
> 1000 Hilltop Circle
> Baltimore MD 21250
> p: 410-455-1943 f:410-455-1042
> e: address@hidden
> physics.umbc.edu/~hoff
> 
> 

Ticket Details
===================
Ticket ID: VFW-525515
Department: Support GEMPAK
Priority: Normal
Status: Open