Web site gets down - errors in mod_jk(apache + tomcat) logs

StackOverflow https://stackoverflow.com/questions/4176433

  •  09-10-2019
  •  | 
  •  

Question

I am having down time problem with my web site. I have apache 2 + mod_jk + tomcat 5.5 setup on Linux Redhat. Just recently started having my website down once a day. Only after restarting tomcat the web site gets back to normal.

Application error log on tomcat(sorry, privacy info hidden in the logs):

2010-11-14 00:40:13,726 ERROR org.apache.catalina.core.StandardWrapperValve [invoke] - Servlet.service() for servlet action threw exception
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:574)
    at ******.exchange.adapter.dispatcher.impl.XMLCommandDispatcher.sendCommand(XMLCommandDispatcher.java:109)
    at ******.exchange.adapter.star******.api.impl.******ApiImpl.******PricingAvailability(******ApiImpl.java:356)
    at ******.exchange.adapter.star******.impl.Concrete******GatewayImpl.find******Promotion(Concrete******GatewayImpl.java:123)
    at ******.exchange.service.impl.Concrete******ManagerImpl.find******Promotion(Concrete******ManagerImpl.java:199)
    at sun.reflect.GeneratedMethodAccessor1142.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:282)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:155)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:122)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:56)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174)
    at $Proxy16.find******Promotion(Unknown Source)
    at ******.ibe.servlet.crdms.search.******SearchPromotionServlet******.handleRequest(******SearchPromotionServlet******.java:75)
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:595)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:543)
    at org.springframework.web.servlet.FrameworkServlet.serviceWrapper(FrameworkServlet.java:390)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:341)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    at java.lang.Thread.run(Thread.java:595)
2010-11-14 00:40:43,762 ERROR org.apache.catalina.core.ApplicationDispatcher [invoke] - Servlet.service() for servlet jsp threw exception
2010-11-14 00:40:43,805 ERROR org.apache.catalina.core.StandardWrapperValve [invoke] - Servlet.service() for servlet jsp threw exception
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:574)
    at java.util.Timer.(Timer.java:154)
    at java.util.Timer.(Timer.java:122)
    at com.mysql.jdbc.Connection.(Connection.java:1441)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
    at java.sql.DriverManager.getConnection(DriverManager.java:525)
    at java.sql.DriverManager.getConnection(DriverManager.java:171)
    at org.apache.jsp.frameset_005f******_005fbottom_005f1_jsp._jspService(frameset_005f******_005fbottom_005f1_jsp.java:88)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:331)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:329)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    at java.lang.Thread.run(Thread.java:595)
2010-11-14 01:54:25,819 ERROR org.apache.catalina.core.ApplicationDispatcher [invoke] - Servlet.service() for servlet jsp threw exception

Mod_jk log entries at time when tomcat gets down:

[Sun Nov 14 00:59:52 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 00:59:52 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:01 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:01 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:01 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed
[Sun Nov 14 01:00:01 2010] worker1 mydomain.com 271.622400

[Sun Nov 14 01:00:01 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:01 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:06 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:06 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:06 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed
[Sun Nov 14 01:00:06 2010] worker1 mydomain.com 86.999946

[Sun Nov 14 01:00:06 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:06 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:10 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:10 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:10 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed
[Sun Nov 14 01:00:10 2010] worker1 mydomain.com 39.000187

Meesages in /var/log/messages:

Nov 14 00:59:40 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:00:47 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:01:49 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:02:54 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:04:03 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:05:06 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:06:13 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:07:14 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:08:15 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:09:19 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:10:23 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:11:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:12:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:13:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:14:56 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:16:04 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:17:19 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:18:22 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:19:26 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:20:26 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:21:36 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:22:37 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:23:48 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:24:50 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:25:54 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:26:57 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:29:08 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:30:10 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "getattr" to /var/lib/mysql/mysql.sock (var_lib_t). For complete SELinux messages. run sea$
Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "getattr" to /var/lib/mysql/nemo.me-tech.com.my.err (var_lib_t). For complete SELinux mess$
Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "read" to ./time_zone_name.frm (var_lib_t). For complete SELinux messages. run sealert -l $

Apache conf:


# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
 ‹IfModule prefork.c›
StartServers         5
MinSpareServers      5
MaxSpareServers     10
MaxClients          256
MaxRequestsPerChild  0
 ‹/IfModule›

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
 ‹IfModule worker.c›
StartServers         2
MaxClients         256
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
 ‹/IfModule›

Apache Mod_jk worker.properties:

 worker.list=worker1
 worker.worker1.type=ajp13
 worker.worker1.host=127.0.0.1
 worker.worker1.port=8009
 worker.worker1.lbfactor=1
 worker.worker1.socket_keepalive=1
 worker.worker1.recycle_timeout=180
 worker.worker1.sticky_session=False

Tomcat server.xml(AJP connector for mod_jk):

   ‹Connector className="org.apache.coyote.tomcat5.CoyoteConnector"
               port="8009"
               protocol="AJP/1.3"
               enableLookups="true"
               redirectPort="8443"
               acceptCount="10"
               debug="0"
               connectionTimeout="5000"
               useURIValidationHack="false"
               maxPostSize="10000000"
               maxThreads="256"
               maxSpareThreads="10"
               URIEncoding="UTF-8"
               protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler"/›

Any help or advise would be highly appreciated.

Thanks a lot

Was it helpful?

Solution

According to this you should try setting your JAVA_OPTS environment variable to reduce the stack and/or heap size used by Tomcat to give the OS some breathing room.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top