Java performance error - Tomcat WebappClassLoader blocked

I ran into this strange problem in the last few days while performing a performance test of my product, I am using java 6, frameworks 3 framework and tomcat 7 server

During a performance test, we start downloading thousands of user interface requests that hit the server, and it works fine, but after a couple of hours the requests begin to block, followed by a surge in CPU usage up to 100%, and even the user interface becomes inaccessible. I took stream dumps to analyze the problem below, this is what I get sequentially.

""http-nio-8443"-exec-1305" daemon prio=10 tid=0x00007f0458538000 nid=0x56dd waiting for monitor entry [0x00007f04345c9000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1663) - locked <0x00000000a162f0c0> (a org.apache.catalina.loader.WebappClassLoader) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1521) at java.beans.Introspector.instantiate(Introspector.java:1448) at java.beans.Introspector.findExplicitBeanInfo(Introspector.java:431) at java.beans.Introspector. (Introspector.java:380) at java.beans.Introspector.getBeanInfo(Introspector.java:232) at java.beans.Introspector.getBeanInfo(Introspector.java:218) at com.googlecode.jsonplugin.JSONWriter.bean(JSONWriter.java:169) at com.googlecode.jsonplugin.JSONWriter.process(JSONWriter.java:152) at com.googlecode.jsonplugin.JSONWriter.value(JSONWriter.java:120) at com.googlecode.jsonplugin.JSONWriter.write(JSONWriter.java:88) at com.googlecode.jsonplugin.JSONUtil.serialize(JSONUtil.java:90) at com.googlecode.jsonplugin.JSONResult.execute(JSONResult.java:119) at com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:348) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253) at com.facetime.imauditor.sreach.action.CustomRequestInterceptorJson.intercept(CustomRequestInterceptorJson.java:69) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:221) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:150) at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:48) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:123) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:167) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:105) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:83) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:207) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:74) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:127) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at org.apache.struts2.interceptor.ProfilingActivationInterceptor.intercept(ProfilingActivationInterceptor.java:107) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:206) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:115) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:143) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:121) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:170) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:123) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:50) at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:504) at org.apache.struts2.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java:419) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.facetime.imcoreserver.registration.AbsoluteSendRedirectFilter.doFilter(AbsoluteSendRedirectFilter.java:48) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:403) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:369) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:317) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1532) - locked <0x00000000abb53778> (a org.apache.tomcat.util.net.SecureNioChannel) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 

And below is the thread statistics.

 Threads locking monitor=1 Threads sleeping on monitor=0 Threads waiting to lock monitor=1097 

Any help in this regard would be really helpful, thanks in advance.

+4
source share
4 answers

this is a bug in struts2 json plugin:

https://issues.apache.org/jira/browse/WW-3902

+3
source

UPDATE:

I recently ran into a similar problem. Sometimes, when a web application is poorly programmed, it may not properly clean the network sockets (TCP) ... for example, database connection classes. Promoting too many open file systems on a host (Tomcat JVM). And this can lead to the fact that you are responsible.

So, in my case, I solved the problem by creating a connection to the database over the stream using generic. This is managed in my bean web application service .:

 private static ThreadLocal<YourConnectionClass> connection = new ThreadLocal<YourConnectionClass>(); public static YourConnectionClass getConnection(){ if (connection.get() == null) connection.set(driver.getConnection()); return connection.get(); } 

This ensures that only one TCP socket will be open in the tomcat executor thread. If this problem applies to you, you will add a similar fix to the one I mentioned, your problems will go away. well, but it's still just a “hunch”. You can check if this applies to you by simply entering this in the unix / linux server console:

 netstat | grep <YOUR_TCP_PORT> 

or using lsof (list of open files). Because TCP sockets are treated as open file descriptors.

 lsof <YOUR_JVM_PID> 

If the number of open sockets after starting tomcat is less or less, and on the other hand, is destructive when there is a problem with your responsibility, then you may have a leak in the handle of the open socket file.

As a second assumption, you can also have a HashTable memory leak, which can be caused even in Java if you incorrectly clean the HashTable of Java objects (somewhere in the web application server code). But, as I can read the comments above, perhaps this is not so.

amuses

will be


My older answer is:

I set my garbage collection strategy.

Try using another by installing the correct OPTS JVM:

 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:SurvivorRatio=2 -XX:NewRatio=8 

Check links:

http://www.petefreitag.com/articles/gctuning/

http://www.idevelopment.info/data/Programming/java/miscellaneous_java/Java_Options_For_Garbage_Collection.html

+1
source

I found a similar problem due to synchronization on WebappClassLoader; appears at any time when there are a large number of loadClass calls, which, unfortunately, include many third-party libraries. The ideal solution is not to call loadClass all the time, however this may not be an option.

This is due to a change made to Tomcat as part of error 48903/44041/48694 to fix the deadlock that some users experienced.

http://svn.apache.org/viewvc?view=revision&revision=927565

In my opinion, the fix is ​​excessively pessimistic for synchronization, I hope I can make a decision that will allow checking the local cache before entering the monitor; which must go.

0
source

The problem can be fixed by upgrading to Tomcat 8.0.16 or higher and installing the Loader class on org.apache.catalina.loader.ParallelWebappClassLoader.

In [TOMCAT_HOME] /conf/context.xml add the following line and restart

<Loader loaderClass="org.apache.catalina.loader.ParallelWebappClassLoader" />

We had a similar problem when Hibernate and Spring JDBC tried to convert a db result set string to an object. This blocking behavior has greatly affected the work of cats with a large number of threads with large result sets.

I would recommend users working with Hibernate and the Spring JDBC application on Tomcat7 with Tomcat 8 with ParallelWebappClassLoader enabled for better performance.

0
source

Source: https://habr.com/ru/post/1437816/


All Articles