[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[AWIPS #KZV-890448]: Unable to connect CAVE clients to local EDEX instance post-upgrade to 16.2.2

Okay we are getting close.  The request log shows repeating messages that
indicate the GFE server is not starting, and that is preventing the request
service from starting fully and accepting connections.


INFO  2017-01-09 19:43:07,010 [pool-3-thread-1] GFESiteActivation: EDEX -
Activation task in progress by another EDEX instance.  Waiting...
INFO  2017-01-09 19:43:17,011 [pool-3-thread-1] GFESiteActivation: EDEX -
Activation task in progress by another EDEX instance.  Waiting...
INFO  2017-01-09 19:43:27,012 [pool-3-thread-1] GFESiteActivation: EDEX -
Activation task in progress by another EDEX instance.  Waiting...


I'm afraid that the next suggestion I have may not work, since you said
that you have already removed and reinstalled the awips2-server group
already, which means there is no chance that old packages or files are
sitting around.

But the message

INFO  2017-01-09 19:20:37,821 [EDEXMain] AppsDefaults: No site
Apps_defaults file found.

indicates that the package awips2-hydroapps-shared may not be installed.
Can you check with the command

rpm -qa |grep awips2-hydroapps-shared

or better yet, send the output of

rpm -qa | grep awips2 | sort

Michael James
Unidata Program Center
Boulder, CO

On Mon, Jan 9, 2017 at 12:47 PM, Clark Evans <address@hidden
> wrote:

> New Client Reply: Unable to connect CAVE clients to local EDEX instance
> post-upgrade to 16.2.2
>
> Hi Michael,
>
>
> I removed ~/caveData and attempted to connect. As before, no luck to the
> local server, but I can connect to edex-cloud.unidata.ucar.edu.
>
>
> Attached are the edex-request-20170109.log and pg_hba.conf files.
>
>
> -Clark
>
>
> address@hidden<mailto:address@hidden>>
> wrote:
>  I'm also wondering if this is an issue of localization incompatibilities.
> You did mention that you could connect to edex-cloud.unidata.ucar.edu<ht
> tp://edex-cloud.unidata.ucar.edu>.  Did
> you try connecting after removing ~/caveData ? If not, could you try?  I
> would recommend *moving* the caveData directory in case there are files you
> want to retain.
>
> mv caveData caveData_old
>
> and then run CAVE again and try to connect.
>
> Michael James
> Unidata Program Center
> Boulder, CO
>
> On Mon, Jan 9, 2017 at 12:34 PM, Michael James <address@hidden<mailto:
> address@hidden>> wrote:
>
> > Thanks for sending the log files, you're right they are not telling us
> > anything.  I do not think that the infrequent error messages in qpid.log
> > are related.
> >
> >  Can you please send me these two files:
> >
> > /awips2/edex/logs/edex-request-20170109.log
> > /awips2/data/pg_hba.conf
> >
> > Michael James
> > Unidata Program Center
> > Boulder, CO
> >
> > On Mon, Jan 9, 2017 at 12:26 PM, Clark Evans <address@hidden.
> > edu> wrote:
> >
> >> New Client Reply: Unable to connect CAVE clients to local EDEX instance
> >> post-upgrade to 16.2.2
> >>
> >> Hi Michael,
> >>
> >> In the qpid.log file, there are a lot of entries of the sort "Resetting
> >> statistics for counter" re: data and messages received, which I presume
> are
> >> normal. However, once each hour, there are messages of the sort listed
> >> below. (There are no telling error messages in the
> >> /awips2/httpd_pypies/var/log/httpd/ error logs, for what it's worth.
> >> I've attached copies of each to this message.)
> >>
> >> Seeing Java mentioned in qpid.log and the site validation menu, I used
> >> yum to update the server to the latest available version of the Java
> >> packages, but this had no effect.
> >>
> >> Best regards,
> >> -Clark
> >>
> >>
> >> 2017-01-09 12:11:19,684 ERROR [IoReceiver - /0:0:0:0:0:0:0:1:50698]
> >> (Logger.java:78) - exception: Connection is not open.
> >> org.apache.qpid.transport.ConnectionException: Connection is not open.
> >>     at org.apache.qpid.transport.Connection.closeCode(Connection.
> >> java:571)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelega
> >> te.connectionClose(ServerConnectionDelegate.java:166)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelega
> >> te.connectionClose(ServerConnectionDelegate.java:56)
> >>     at org.apache.qpid.transport.ConnectionClose.dispatch(Connectio
> >> nClose.java:91)
> >>     at org.apache.qpid.transport.ConnectionDelegate.control(Connect
> >> ionDelegate.java:49)
> >>     at org.apache.qpid.transport.ConnectionDelegate.control(Connect
> >> ionDelegate.java:40)
> >>     at org.apache.qpid.transport.Method.delegate(Method.java:163)
> >>     at org.apache.qpid.transport.Connection.received(
> Connection.java:405)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection.
> >> access$001(ServerConnection.java:64)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection$1.
> >> run(ServerConnection.java:316)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection$1.
> >> run(ServerConnection.java:312)
> >>     at java.security.AccessController.doPrivileged(Native Method)
> >>     at javax.security.auth.Subject.doAs(Subject.java:356)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection.
> >> received(ServerConnection.java:311)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection.
> >> received(ServerConnection.java:64)
> >>     at org.apache.qpid.transport.network.Assembler.emit(Assembler.
> >> java:97)
> >>     at org.apache.qpid.transport.network.Assembler.assemble(Assembl
> >> er.java:183)
> >>     at org.apache.qpid.transport.network.Assembler.frame(Assembler.
> >> java:131)
> >>     at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128)
> >>     at org.apache.qpid.transport.network.Assembler.received(Assembl
> >> er.java:102)
> >>     at org.apache.qpid.transport.network.Assembler.received(Assembl
> >> er.java:44)
> >>     at org.apache.qpid.transport.network.InputHandler.next(InputHan
> >> dler.java:199)
> >>     at org.apache.qpid.transport.network.InputHandler.received(Inpu
> >> tHandler.java:114)
> >>     at org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10.
> >> received(ProtocolEngine_0_10.java:179)
> >>     at org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10.
> >> received(ProtocolEngine_0_10.java:43)
> >>     at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.r
> >> eceived(MultiVersionProtocolEngine.java:153)
> >>     at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.r
> >> eceived(MultiVersionProtocolEngine.java:51)
> >>     at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiv
> >> er.java:161)
> >>     at java.lang.Thread.run(Thread.java:745)
> >> 2017-01-09 12:11:19,685 ERROR [IoReceiver - /0:0:0:0:0:0:0:1:50698]
> >> (Logger.java:78) - exception: sender for socket /0:0:0:0:0:0:0:1:50698
> is
> >> closed
> >> org.apache.qpid.transport.ConnectionException: sender for socket
> >> /0:0:0:0:0:0:0:1:50698 is closed
> >>     at org.apache.qpid.transport.Connection.exception(Connection.
> >> java:561)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection.
> >> exception(ServerConnection.java:274)
> >>     at org.apache.qpid.transport.network.Assembler.exception(Assemb
> >> ler.java:107)
> >>     at org.apache.qpid.transport.network.InputHandler.exception(
> >> InputHandler.java:209)
> >>     at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.e
> >> xception(MultiVersionProtocolEngine.java:158)
> >>     at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiv
> >> er.java:217)
> >>     at java.lang.Thread.run(Thread.java:745)
> >> Caused by: org.apache.qpid.transport.SenderClosedException: sender for
> >> socket /0:0:0:0:0:0:0:1:50698 is closed
> >>     at org.apache.qpid.transport.network.io.IoSender.checkNotAlread
> >> yClosed(IoSender.java:382)
> >>     at org.apache.qpid.transport.network.io.IoSender.send(IoSender.
> >> java:120)
> >>     at org.apache.qpid.transport.network.io.IoSender.send(IoSender.
> >> java:40)
> >>     at org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10$2.
> >> send(ProtocolEngine_0_10.java:114)
> >>     at org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10$2.
> >> send(ProtocolEngine_0_10.java:102)
> >>     at org.apache.qpid.transport.network.Disassembler.frame(Disasse
> >> mbler.java:117)
> >>     at org.apache.qpid.transport.network.Disassembler.fragment(Disa
> >> ssembler.java:147)
> >>     at org.apache.qpid.transport.network.Disassembler.method(Disass
> >> embler.java:233)
> >>     at org.apache.qpid.transport.network.Disassembler.control(Disas
> >> sembler.java:167)
> >>     at org.apache.qpid.transport.network.Disassembler.control(Disas
> >> sembler.java:48)
> >>     at org.apache.qpid.transport.Method.delegate(Method.java:163)
> >>     at org.apache.qpid.transport.network.Disassembler.send(Disassem
> >> bler.java:73)
> >>     at org.apache.qpid.transport.network.Disassembler.send(Disassem
> >> bler.java:48)
> >>     at org.apache.qpid.transport.Connection.send(Connection.java:420)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection.send(
> >> ServerConnection.java:558)
> >>     at org.apache.qpid.transport.Connection.invoke(Connection.java:439)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection.
> >> invoke(ServerConnection.java:128)
> >>     at org.apache.qpid.transport.ConnectionInvoker.connectionCloseO
> >> k(ConnectionInvoker.java:73)
> >>     at org.apache.qpid.transport.ConnectionDelegate.sendConnectionC
> >> loseOkAndCloseSender(ConnectionDelegate.java:81)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelega
> >> te.connectionClose(ServerConnectionDelegate.java:168)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnectionDelega
> >> te.connectionClose(ServerConnectionDelegate.java:56)
> >>     at org.apache.qpid.transport.ConnectionClose.dispatch(Connectio
> >> nClose.java:91)
> >>     at org.apache.qpid.transport.ConnectionDelegate.control(Connect
> >> ionDelegate.java:49)
> >>     at org.apache.qpid.transport.ConnectionDelegate.control(Connect
> >> ionDelegate.java:40)
> >>     at org.apache.qpid.transport.Method.delegate(Method.java:163)
> >>     at org.apache.qpid.transport.Connection.received(
> Connection.java:405)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection.
> >> access$001(ServerConnection.java:64)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection$1.
> >> run(ServerConnection.java:316)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection$1.
> >> run(ServerConnection.java:312)
> >>     at java.security.AccessController.doPrivileged(Native Method)
> >>     at javax.security.auth.Subject.doAs(Subject.java:356)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection.
> >> received(ServerConnection.java:311)
> >>     at org.apache.qpid.server.protocol.v0_10.ServerConnection.
> >> received(ServerConnection.java:64)
> >>     at org.apache.qpid.transport.network.Assembler.emit(Assembler.
> >> java:97)
> >>     at org.apache.qpid.transport.network.Assembler.assemble(Assembl
> >> er.java:183)
> >>     at org.apache.qpid.transport.network.Assembler.frame(Assembler.
> >> java:131)
> >>     at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128)
> >>     at org.apache.qpid.transport.network.Assembler.received(Assembl
> >> er.java:102)
> >>     at org.apache.qpid.transport.network.Assembler.received(Assembl
> >> er.java:44)
> >>     at org.apache.qpid.transport.network.InputHandler.next(InputHan
> >> dler.java:199)
> >>     at org.apache.qpid.transport.network.InputHandler.received(Inpu
> >> tHandler.java:114)
> >>     at org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10.
> >> received(ProtocolEngine_0_10.java:179)
> >>     at org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10.
> >> received(ProtocolEngine_0_10.java:43)
> >>     at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.r
> >> eceived(MultiVersionProtocolEngine.java:153)
> >>     at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.r
> >> eceived(MultiVersionProtocolEngine.java:51)
> >>     at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiv
> >> er.java:161)
> >>     ... 1 more
> >>
> >>
> >> address@hidden<mailto:address@hidden
> ><mailto:address@hidden<mailto:
> address@hidden>>>
> >> wrote:
> >> > Thanks for the quick response. Unfortunately (or fortunately?), I have
> >> had the fully-qualified EDEX server name in /awips2/edex/bin/setup.env
> >> (also attached):
> >> >
> >> > export EDEX_SERVER=awips2.math.uwm.edu<http://awips2.math.uwm.edu><
> http://awips2.math.uwm.edu>
> >>
> >>
> >> Okay, this is strange.  The next thing to check would be the qpid log
> >> file /awips2/qpid/logs/qpid.log to see if there are any telling error
> >> messages.  If not, I would then check the most recent error log file in
> >> /awips2/httpd_pypies/var/log/httpd/ for messages.
> >>
> >> -Michael
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: KZV-890448
> >> Department: Support AWIPS
> >> Priority: Normal
> >> Status: Open
> >> ===================
> >> NOTE: All email exchanges with Unidata User Support are recorded in the
> >> Unidata inquiry tracking system and then made publicly available through
> >> the web.  If you do not want to have your interactions made available in
> >> this way, you must let us know in each email you send to us.
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Clark Evans
> >> Associate Professor and Program Coordinator
> >> Atmospheric Science Program
> >> Department of Mathematical Sciences
> >> University of Wisconsin-Milwaukee
> >> E-mail: address@hidden<mailto:address@hidden><mailto:address@hidden
> <mailto:address@hidden>>
> >> Phone: (414) 229-4469
> >> Web:    http://derecho.math.uwm.edu/
> >>
> >>
> >>
> >> Clark Evans
> >> Associate Professor and Program Coordinator
> >> Atmospheric Science Program
> >> Dept. of Mathematical Sciences
> >> University of Wisconsin-Milwaukee
> >> E-mail:     address@hidden<mailto:address@hidden>
> >> Phone:     (414) 229-4469
> >> Web:        http://derecho.math.uwm.edu/
> >>
> >>
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: KZV-890448
> >> Department: Support AWIPS
> >> Priority: Normal
> >> Status: Open
> >> Link:  https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=ti
> >> ckets&_a=viewticket&ticketid=27839
> >>
> >
> >
>
>
>
> Ticket Details
> ===================
> Ticket ID: KZV-890448
> Department: Support AWIPS
> Priority: Normal
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publicly available through
> the web.  If you do not want to have your interactions made available in
> this way, you must let us know in each email you send to us.
>
>
>
>
>
> --
> Clark Evans
> Associate Professor and Program Coordinator
> Atmospheric Science Program
> Department of Mathematical Sciences
> University of Wisconsin-Milwaukee
> E-mail: address@hidden<mailto:address@hidden>
> Phone: (414) 229-4469
> Web:    http://derecho.math.uwm.edu/
>
>
>
> Clark Evans
> Associate Professor and Program Coordinator
> Atmospheric Science Program
> Dept. of Mathematical Sciences
> University of Wisconsin-Milwaukee
> E-mail:     address@hidden
> Phone:     (414) 229-4469
> Web:        http://derecho.math.uwm.edu/
>
>
>
> Ticket Details
> ===================
> Ticket ID: KZV-890448
> Department: Support AWIPS
> Priority: Normal
> Status: Open
> Link:  https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=
> tickets&_a=viewticket&ticketid=27839



Ticket Details
===================
Ticket ID: KZV-890448
Department: Support AWIPS
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.