[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #JVF-237504]: 6.11.4 segfault on RHEL6.4 64bit



Daryl,

[Sorry for taking so long to respond. I was in an AWIPS-II administration 
workshop all week.]

Was the LDM version 6.11.4?

The core-dump indicates that the binary is newer than the core file. This is 
bad because the stack trace can't, therefore, be trusted.

Is there any chance I could log onto Iem21 as the LDM user?

> I attempted to use 6.11.x for the first time today.
> 
> It appeared to build
> just fine on my RHEL6.4 64bit system, but it immediately segfaulted after
> startup.  I've added the log file.
> 
> The server logged message:
> 
> ldmd[18269]: segfault at 0 ip 00002b1b6f067376 sp 00007fff43fed6d0 error 6
> in libldm.so.0.0.0[2b1b6f035000+53000]
> 
> abrtd captured a core dump of the process:
> 
> address@hidden ccpp-2013-04-08-13:44:28-14429]# gdb /home/ldm/bin/ldmd
> coredump
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /home/ldm/bin/ldmd...done.
> 
> warning: exec file is newer than core file.
> 
> [New Thread 14429]
> Missing separate debuginfo for /home/ldm/lib/libldm.so.0
> Try: yum --disablerepo='*' --enablerepo='*-debug*' install
> /usr/lib/debug/.build-id/f9/eb16e0c49470b9ad4b3549ec8a258d6d86238d
> Missing separate debuginfo for /home/ldm/lib/libxml2.so.2
> Try: yum --disablerepo='*' --enablerepo='*-debug*' install
> /usr/lib/debug/.build-id/c5/78910b23d719a098d7d2b181e1545052aeb58d
> Missing separate debuginfo for
> Try: yum --disablerepo='*' --enablerepo='*-debug*' install
> /usr/lib/debug/.build-id/a6/993d9af0d108bfc4a2bbfdb176ea3288f6fd5c
> Reading symbols from /home/ldm/lib/libldm.so.0...done.
> Loaded symbols for /home/ldm/lib/libldm.so.0
> Reading symbols from /home/ldm/lib/libxml2.so.2...done.
> Loaded symbols for /home/ldm/lib/libxml2.so.2
> Reading symbols from /lib64/libdl.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libm.so.6...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
> found)...done.
> [Thread debugging using libthread_db enabled]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libc.so.6...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libnss_files.so.2
> Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libnss_dns.so.2
> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libresolv.so.2
> Core was generated by `ldmd -I 0.0.0.0 -P 388 -M 256 -m 3600 -o 3600 -q
> /home/ldm/var/queues/ldm.pq /h'.
> Program terminated with signal 11, Segmentation fault.
> #0  sm_append (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0)
> at uldb.c:1225
> 1225          eps->size = nbytes;
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.12-1.107.el6.x86_64
> (gdb) bt full
> #0  sm_append (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0)
> at uldb.c:1225
> prodSpec = 0x20bd1e0
> pattern = <value optimized out>
> nbytes = <value optimized out>
> entrySize = 88
> segment = 0x2b8107f5b000
> i = <value optimized out>
> entry = 0x2b8107f5c358
> epc = <value optimized out>
> eps = <value optimized out>
> numProdSpecs = 2
> prodSpecsSize = <value optimized out>
> #1  sm_addUpstreamLdm (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0) at uldb.c:1278
> status = 0
> sm = 0x2b81078c59b0
> #2  sm_add (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0)
> at uldb.c:1409
> sub = 0x20bd160
> status = <value optimized out>
> #3  uldb_addProcess (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0)
> at uldb.c:1982
> sub = 0x0
> status = <value optimized out>
> #4  0x00002b8107680dcb in forn_5_svc (want=0x7fffcd45c100,
> rqstp=0x7fffcd45c6a0, ident=0x2b81076a2d1f "(feed)", doit=0x2b8107681960
> <feed5_sqf>)
> at forn5_svc.c:401
> theReply = {code = OK, ldm_replyt_u = {dpktdp = 0x0, signaturep =
> 0x0, alternatep = 0x0, newclssp = 0x0}}
> hc = {state = 749,
> remote =
> "\001\000\000\000H\300E\315\377\177\000\000\000\000\000\000\000\000\000\000
> address@hidden>~",
> prog = 227547323962, vers = 0, prot = 0, addr = {sin_family = 0,
> sin_port = 0, sin_addr = {s_addr = 1}, sin_zero =
> "4\000\000\000\000\000\000"}, pmap_clnt = 0x1, rpcerr = {ru = {RE_errno =
> 124114632,
> RE_why = 124114632, RE_vers = {low = 47833174890184, high =
> 0}, RE_lb = {s1 = 47833174890184, s2 = 0}}, re_status = 124115488},
> port = 49104, clnt = 0x100000000, h_timeo = 0, begin = {tv_sec =
> 47833174891040, tv_usec = 140736637288432}, elapsed = {tv_sec = 34227772,
> tv_usec = 140736637288456},
> errmsg =
> "\206K(\373\001\000\000\000\300\353\t\002\000\000\000\000}Qf\a\201+\000\000\024\000\000\000\000\000\000\000Ãi\a\201+\000\000\300\353\t\---Type
> <return> to continue, or q <return> to quit---
> 002\000\000\000\000\024\000\000\000\000\000\000\000P\274\v\002\000\000\000\000\000\301E\315\377\177\000\000\240\306E\315\377\177\000\000\240\306E\315\377\177\000\000\220Vh\a\201+\000\000\355&j\a\201+\000\000\340\314h\a\201+\000\000\366\322i\a\201+\000\000\370;f\a\201+\000\000\310\326e\a\201+\000\000\020\300E\315\377\177\000\000\377\377\377\377\000\000\000\000P\274\v\002\000\000\000\000\001\000\000\000\000\000\000\000\330\065%",
> '\000' <repeats 13 times>"\355,
> &j\a\201+\000\000\340\314h\a\201+\000\000\000\340e\a\201+\000\000\320\337\340\372\064\000\000\000\001",
> '\000' <repeats 15 times>"\360,
> \351\t\002\000\000\000\000\370;f\a\201+\000\000(\301E\315\377\177\000"}
> rpc_stat = <value optimized out>
> status = <value optimized out>
> mt = TV_GT
> lastsent = {tv_sec = 47833175053968, tv_usec = 227547367061}
> now = {tv_sec = 140736637288704, tv_usec = 140736637290144}
> keepalive_interval = 300
> remote = 0x2b81078c5640
> uldbSub = 0x0
> downAddr = <value optimized out>
> #5  0x00002b8107684aa0 in ldmprog_5 (rqstp=0x7fffcd45c6a0,
> transp=0x209e980) at ldm5_svc.c:109
> argument = {hereis_5_arg = {info = {arrival = {tv_sec =
> 1365446433, tv_usec = 22227},
> signature =
> "\377\377\377\177\000\000\000\000?B\017\000\000\000\000", origin = 0x2
> <Address 0x2 out of bounds>, feedtype = 34323328,
> seqno = 0, ident = 0x0, sz = 0}, data = 0x0}, feedme_5_arg =
> {from = {tv_sec = 1365446433, tv_usec = 22227}, to = {tv_sec = 2147483647,
> tv_usec = 999999}, psa = {psa_len = 2, psa_val =
> 0x20bbb80}}, hiya_5_arg = {from = {tv_sec = 1365446433, tv_usec = 22227},
> to = {
> tv_sec = 2147483647, tv_usec = 999999}, psa = {psa_len = 2,
> psa_val = 0x20bbb80}}, notification_5_arg = {arrival = {tv_sec =
> 1365446433,
> tv_usec = 22227}, signature =
> "\377\377\377\177\000\000\000\000?B\017\000\000\000\000", origin = 0x2
> <Address 0x2 out of bounds>,
> feedtype = 34323328, seqno = 0, ident = 0x0, sz = 0},
> notifyme_5_arg = {from = {tv_sec = 1365446433, tv_usec = 22227}, to = {
> tv_sec = 2147483647, tv_usec = 999999}, psa = {psa_len = 2,
> psa_val = 0x20bbb80}}, comingsoon_5_arg = {infop = 0x51630f21, pktsz =
> 22227},
> blkdata_5_arg = {signaturep = 0x51630f21, pktnum = 22227, data =
> {dbuf_len = 2147483647, dbuf_val = 0xf423f <Address 0xf423f out of
> bounds>}}}
> result = <value optimized out>
> xdr_argument = 0x2b8107685690 <xdr_prod_class>
> xdr_result = 0x2b81076853b0 <xdr_ldm_replyt>
> local = 0x2b810768cce0 <feedme_5_svc>
> procName = 0x2b81076a26ed "FEEDME"
> #6  0x00002b8107699249 in svc_getreqsock (sock=<value optimized out>) at
> svc.c:541
> s = <value optimized out>
> why = <value optimized out>
> stat = <value optimized out>
> msg = {rm_xid = 1365666557, rm_direction = CALL, ru = {RM_cmb =
> {cb_rpcvers = 2, cb_prog = 300029, cb_vers = 5, cb_proc = 4, cb_cred = {
> oa_flavor = 0, oa_base = 0x7fffcd45c190 "\001", oa_length
> = 0}, cb_verf = {oa_flavor = 0, oa_base = 0x7fffcd45c320 "", oa_length =
> 0}},
> RM_rmb = {rp_stat = 2, ru = {RP_ar = {ar_verf = {oa_flavor =
> 300029, oa_base = 0x5 <Address 0x5 out of bounds>, oa_length = 4},
> ---Type <return> to continue, or q <return> to quit---
> ar_stat = SUCCESS, ru = {AR_versions = {low =
> 140736637288848, high = 0}, AR_results = {where = 0x7fffcd45c190 "\001",
> proc = 0}}},
> RP_dr = {rj_stat = 300029, ru = {RJ_versions = {low = 5,
> high = 4}, RJ_why = AUTH_TOOWEAK}}}}}}
> prog_found = <value optimized out>
> low_vers = <value optimized out>
> high_vers = <value optimized out>
> r = {rq_prog = 300029, rq_vers = 5, rq_proc = 4, rq_cred =
> {oa_flavor = 0, oa_base = 0x7fffcd45c190 "\001", oa_length = 0},
> rq_clntcred = 0x7fffcd45c4b0 "P)f\a\201+", rq_xprt = 0x209e980}
> xprt = 0x209e980
> cred_area =
> "\001\000\000\000\201+\000\000\000\000\000\000\000\000\000\000\006\000\000\000\000\000\000\000?
> \373\064\000\000\000\000\313\t\002\000\000\000\000\000\004\000\000\000\000\000\000
> address@hidden",
> '\000' <repeats 15 times>"\240,
> \023Y\373\064\000\000\000\200\300\005\002\000\000\000\000\000\004\000\000\000\000\000\000\000\313\t\002\000\000\000\000\000\004\000\000\000\000\000\000ux\325\a\201+\000\000\330\313E\315\377\177\000\000\023^\325\a\201+\000\000\000\000\000\000\064\000\000\000\065\345\340\372\064\000\000\000\270\002\000\000\000\000\000\000\001",
> '\000' <repeats 15 times>"\320,
> \302E\315\377\177\000\000\002\000\000\000\v\000\000\000
> \303E\315\377\177\000\000\000\313\t\002\000\000\000\000"...
> #7  0x00002b8107689281 in one_svc_run (sock=1, timeout=<value optimized
> out>) at one_svc_run.c:86
> readFds = {fds_bits = {2, 0 <repeats 15 times>}}
> before = {tv_sec = 1365446668, tv_usec = 444911}
> selectStatus = 1
> canonicalTimeout = {tv_sec = 60, tv_usec = 0}
> selectTimeout = {tv_sec = 59, tv_usec = 999997}
> fds = {fds_bits = {2, 0 <repeats 15 times>}}
> #8  0x0000000000406553 in handle_connection (sock=0) at ldmd.c:714
> TIMEOUT = 60
> raddr = {sin_family = 2, sin_port = 38356, sin_addr = {s_addr =
> 2276285068}, sin_zero = "\000\000\000\000\000\000\000"}
> len = 16
> xp_sock = 1
> pid = <value optimized out>
> xprt = 0x209e980
> status = 1
> remote = 0x2b81078c5640
> #9  sock_svc (sock=0) at ldmd.c:773
> ready = <value optimized out>
> readfds = {fds_bits = {1, 0 <repeats 15 times>}}
> stimeo = {tv_sec = 5, tv_usec = 712792}
> width = 1
> #10 0x0000000000406c86 in main (ac=<value optimized out>, av=<value
> optimized out>) at ldmd.c:1047
> ---Type <return> to continue, or q <return> to quit---
> pqfname = 0x7fffcd45df4c "/home/ldm/var/queues/ldm.pq"
> sock = <value optimized out>
> status = <value optimized out>
> doSomething = 1
> locIpAddr = 0
> ldmPort = 388
> 
> I reverted to 6.10.1 and it appears to be happily chugging along again.
> 
> daryl
> 
> --
> /**
> * Daryl Herzmann
> * Assistant Scientist -- Iowa Environmental Mesonet
> * http://mesonet.agron.iastate.edu
> */
> 
Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: JVF-237504
Department: Support LDM
Priority: High
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.