Re: [iu-lead-dev] lead.unidata.ucar.edu:8080 LEAD thredds server unresponsive

[Brian Kelly <bkelly@xxxxxxxxxxxxxxxx>]

Hi Felix,

Interesting.  A restart of the tomcat server has seemed to fix the 
problem, although I am noticing the restarted server does seem very 
busy.  I'll keep looking.

Brian


Felix Terkhorn wrote:
> I did some poking around and found interesting output in the 
> tomcat@xxxxxxxxxxxxxxxxxxxxx logs  (/tomcat/logs/catalina.out) on 
> leadserv
>
> The log is pretty noisy, but I'm excerpting some of the juicy below.
>
> Best regards,
> Felix
>
> <snip>
>
> Error Grib 1 record in Grib2 file
>
> java.lang.OutOfMemoryError: Java heap space
>
> <snip>
>
> DODServlet ERROR (anyExceptionHandler): java.lang.OutOfMemoryError: 
> Java heap space
>
> ReqState:
>
>  serverClassName:    'thredds.server.opendap.NcDODSServlet'
>
>  dataSet:            
> 'LEAD/model/NCEP/LEADNAM/2008100518/nam40grb2.2008100518f60'
>
>  requestSuffix:      'html'
>
>  CE:                 ''
>
>  compressOK:          true
>
>  InitParameters:
>
>    DebugOn: ''
>
> <snip>
>
> 2009-02-28T07:58:05.493 -0700 [ 142684369][ 2448832] WARN  - 
> thredds.servlet.CatalogServicesServlet - Cant find 
> dataset=NCEP/NAM/CONUS_20km/noaaport/NA
>
> M_CONUS_20km_noaaport_20081218_0600.grib1 in 
> catalog=http://lead.unidata.ucar.edu:8080/thredds/catalog/model/NCEP/NAM/CONUS_20km/noaaport/catalog.xml 
>
>
> Error Grib 1 record in Grib2 file
>
> 2009-02-28T07:59:57.999 -0700 [ 142796875][ 2297860] ERROR - 
> thredds.server.opendap.NcDODSServlet - Error
> java.lang.OutOfMemoryError: Java heap space
>
> 2009-02-28T08:00:01.852 -0700 [ 142800728][ 2448788] ERROR - 
> thredds.server.ncSubset.GridServlet - GridServlet.showForm
>
> ClientAbortException:  java.net.SocketException: Broken pipe
>
>        at 
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358) 
>
>
>        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:354)
>
>        at 
> org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:381) 
>
>
>        at 
> org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:370)
>
>        at 
> org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89) 
>
>
>        at 
> org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:83) 
>
>
>        at 
> thredds.server.ncSubset.GridServlet.showForm(GridServlet.java:461)
>
>        at thredds.server.ncSubset.GridServlet.doGet(GridServlet.java:125)
>
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
>
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>
>        at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) 
>
>
>        at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
>
>
>        at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:228) 
>
>
>        at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) 
>
>
>        at 
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433) 
>
>
>        at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) 
>
>
>        at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) 
>
>
>        at 
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:517)
>
>        at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) 
>
>
>        at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:216) 
>
>
>        at 
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) 
>
>
>        at 
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:634) 
>
>
>        at 
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:445)
>
>        at java.lang.Thread.run(Thread.java:619)
>
> Caused by: java.net.SocketException: Broken pipe
>
>        at java.net.SocketOutputStream.socketWrite0(Native Method)
>
>        at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>
>        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>
>        at 
> org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:764) 
>
>
>        at 
> org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:118) 
>
>
>        at 
> org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:570) 
>
>
>        at org.apache.coyote.Response.doWrite(Response.java:560)
>
>        at 
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353) 
>
>
>        ... 23 more
>
> <snip>
>
>
> Terkhorn, Felix M wrote:
> > Hi Brian and Mike, THREDDS team,
> >
> >   It looks like Unidata LEAD THREDDS server at 
> > http://lead.unidata.ucar.edu:8080/thredds/lead/leadcatalog.html   is 
> > not responding again.
> >
> >   In addition to being unable to access the web frontend, our 
> > indexing tool is not able to crosswalk data products
> >  
> >   Can you please check into it?
> > Thanks,
> > Felix
> >
> > _______________________________________________
> > iu-lead-dev mailing list
> > iu-lead-dev@xxxxxxxxxxxxxxxxxxxxxx
> > http://mailman.cs.indiana.edu/mailman/listinfo/iu-lead-dev
> >