Tomcat Guice / JDBC memory leak

Question

I experience a memory leak due to orphan threads in Tomcat. In particular, it seems that Guice and the JDBC driver do not close streams.

Aug 8, 2012 4:09:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: A web application appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak. Aug 8, 2012 4:09:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: A web application appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. 

I know this sounds like other questions (like this one ), but in my case the answer “don't worry about it” won 'enough, as it causes problems for me. I have a CI server that regularly updates this application, and after 6-10 reboots, the CI server will freeze because Tomcat is not working.

I need to clarify these orphaned streams so that I can more confidently launch my CI server. Any help would be appreciated!

Answer 1

I just coped with this problem myself. Contrary to some other answers, I do not recommend issuing the t.stop() command. This method was outdated and not without reason. Link Oracle reasons for this.

However, there is a solution to remove this error without having to resort to t.stop() ...

You can use most of the @Oso code, just replace the next section

 Set<Thread> threadSet = Thread.getAllStackTraces().keySet(); Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]); for(Thread t:threadArray) { if(t.getName().contains("Abandoned connection cleanup thread")) { synchronized(t) { t.stop(); //don't complain, it works } } } 

Replace it using the following method provided by the MySQL driver:

 try { AbandonedConnectionCleanupThread.shutdown(); } catch (InterruptedException e) { logger.warn("SEVERE problem cleaning up: " + e.getMessage()); e.printStackTrace(); } 

This should properly disable the thread, and the error should disappear.

Answer 2

I had the same problem, and as Jeff says, “don’t worry about it” wasn’t.

I made a ServletContextListener that stops the freezing thread when closing the context, and then registered such a ContextListener in the web.xml file.

I already know that stopping the thread is not an elegant way to deal with them, but otherwise the server continues to crash after two or three deployments (it is not always possible to restart the application server).

The class I created:

 public class ContextFinalizer implements ServletContextListener { private static final Logger LOGGER = LoggerFactory.getLogger(ContextFinalizer.class); @Override public void contextInitialized(ServletContextEvent sce) { } @Override public void contextDestroyed(ServletContextEvent sce) { Enumeration<Driver> drivers = DriverManager.getDrivers(); Driver d = null; while(drivers.hasMoreElements()) { try { d = drivers.nextElement(); DriverManager.deregisterDriver(d); LOGGER.warn(String.format("Driver %s deregistered", d)); } catch (SQLException ex) { LOGGER.warn(String.format("Error deregistering driver %s", d), ex); } } Set<Thread> threadSet = Thread.getAllStackTraces().keySet(); Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]); for(Thread t:threadArray) { if(t.getName().contains("Abandoned connection cleanup thread")) { synchronized(t) { t.stop(); //don't complain, it works } } } } } 

After creating the class, register it in the web.xml file:

 <web-app... <listener> <listener-class>path.to.ContextFinalizer</listener-class> </listener> </web-app> 

Answer 3

Starting with version 5.1.23 from MySQL, a method is provided to close the reject connection cleanup thread, AbandonedConnectionCleanupThread.shutdown .

However, we do not need direct dependencies in our code on the otherwise opaque JDBC driver code, so my solution is to use reflection to find the class and method and call it if it is found. The following complete code snippet is all that is needed, executed in the context of the class loader loading the JDBC driver:

 try { Class<?> cls=Class.forName("com.mysql.jdbc.AbandonedConnectionCleanupThread"); Method mth=(cls==null ? null : cls.getMethod("shutdown")); if(mth!=null) { mth.invoke(null); } } catch (Throwable thr) { thr.printStackTrace(); } 

This purely terminates the thread if the JDBC driver is the latest version of the MySQL connector and does nothing otherwise.

Note that it must be executed in the context of the class loader, because the stream is a static link; if the driver class was not or was not yet unloaded when this code was run, the thread will not be launched for subsequent JDBC interactions.

Answer 4

The least invasive workaround is to force the initialization of the MySQL JDBC driver from code outside the webapp class loader.

In tomcat / conf / server.xml change (inside the Server element):

 <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> 

to

 <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" classesToInitialize="com.mysql.jdbc.NonRegisteringDriver" /> 

It is assumed that you install the MySQL JDBC driver in the tomcat lib directory, and not inside your webapp.war WEB-INF / lib directory, since the thing is to load the driver before and regardless of your webapp.

Literature:

• http://bugs.mysql.com/bug.php?id=68556#c400606

• http://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html#JRE_Memory_Leak_Prevention_Listener_-_org.apache.catalina.core.JreMemoryLeakPreventionListener

• http://markmail.org/message/dmvlkps7lbgpngil

Answer 5

I took the best parts of the answers above and combined them into an easily extensible class. This combines the original Oso offering with improved Bill drivers and better reflection of Software Monkey. (I liked the simplicity of Stephan L's answer, but sometimes changing the Tomcat environment itself is not a good option, especially if you have to deal with autoscaling or porting to another web container.)

Instead of directly accessing the class name, thread name, and stop method, I also encapsulated them in ThreadInfo's private inner class. Using a list of these ThreadInfo objects, you can enable additional nasty threads to shut down the same code. This is a bit more complicated than most people who probably need it, but should work more generally when you need it.

 import java.lang.reflect.Method; import java.sql.Driver; import java.sql.DriverManager; import java.sql.SQLException; import java.util.Arrays; import java.util.Enumeration; import java.util.List; import java.util.Set; import javax.servlet.ServletContextEvent; import javax.servlet.ServletContextListener; import org.slf4j.Logger; import org.slf4j.LoggerFactory; /** * Context finalization to close threads (MySQL memory leak prevention). * This solution combines the best techniques described in the linked Stack * Overflow answer. * @see <a href="https://stackoverflow.com/questions/11872316/tomcat-guice-jdbc-memory-leak">Tomcat Guice/JDBC Memory Leak</a> */ public class ContextFinalizer implements ServletContextListener { private static final Logger LOGGER = LoggerFactory.getLogger(ContextFinalizer.class); /** * Information for cleaning up a thread. */ private class ThreadInfo { /** * Name of the thread initiating class. */ private final String name; /** * Cue identifying the thread. */ private final String cue; /** * Name of the method to stop the thread. */ private final String stop; /** * Basic constructor. * @param n Name of the thread initiating class. * @param c Cue identifying the thread. * @param s Name of the method to stop the thread. */ ThreadInfo(final String n, final String c, final String s) { this.name = n; this.cue = c; this.stop = s; } /** * @return the name */ public String getName() { return this.name; } /** * @return the cue */ public String getCue() { return this.cue; } /** * @return the stop */ public String getStop() { return this.stop; } } /** * List of information on threads required to stop. This list may be * expanded as necessary. */ private List<ThreadInfo> threads = Arrays.asList( // Special cleanup for MySQL JDBC Connector. new ThreadInfo( "com.mysql.jdbc.AbandonedConnectionCleanupThread", //$NON-NLS-1$ "Abandoned connection cleanup thread", //$NON-NLS-1$ "shutdown" //$NON-NLS-1$ ) ); @Override public void contextInitialized(final ServletContextEvent sce) { // No-op. } @Override public final void contextDestroyed(final ServletContextEvent sce) { // Deregister all drivers. Enumeration<Driver> drivers = DriverManager.getDrivers(); while (drivers.hasMoreElements()) { Driver d = drivers.nextElement(); try { DriverManager.deregisterDriver(d); LOGGER.info( String.format( "Driver %s deregistered", //$NON-NLS-1$ d ) ); } catch (SQLException e) { LOGGER.warn( String.format( "Failed to deregister driver %s", //$NON-NLS-1$ d ), e ); } } // Handle remaining threads. Set<Thread> threadSet = Thread.getAllStackTraces().keySet(); Thread[] threadArray = threadSet.toArray(new Thread[threadSet.size()]); for (Thread t:threadArray) { for (ThreadInfo i:this.threads) { if (t.getName().contains(i.getCue())) { synchronized (t) { try { Class<?> cls = Class.forName(i.getName()); if (cls != null) { Method mth = cls.getMethod(i.getStop()); if (mth != null) { mth.invoke(null); LOGGER.info( String.format( "Connection cleanup thread %s shutdown successfully.", //$NON-NLS-1$ i.getName() ) ); } } } catch (Throwable thr) { LOGGER.warn( String.format( "Failed to shutdown connection cleanup thread %s: ", //$NON-NLS-1$ i.getName(), thr.getMessage() ) ); thr.printStackTrace(); } } } } } } } 

Answer 6

I went further from Oso, improved the code above at two points:

• Finalizer stream added to check for need:

   for(Thread t:threadArray) { if(t.getName().contains("Abandoned connection cleanup thread") || t.getName().matches("com\\.google.*Finalizer") ) { synchronized(t) { logger.warn("Forcibly stopping thread to avoid memory leak: " + t.getName()); t.stop(); //don't complain, it works } } } 

• Sleep a bit to give time to stop. Without this, tomcat continued to complain.

   try { Thread.sleep(1000); } catch (InterruptedException e) { logger.debug(e.getMessage(), e); } 

Answer 7

Bill's solution looks good, however I found another solution directly in the MySQL error reports:

[June 5, 2013 17:12] Christopher Schulz There is a much better workaround here until something changes.

Enable Tomcat JreMemoryLeakPreventionListener (enabled by default on Tomcat 7) and add this attribute to the element:

classesToInitialize = "com.mysql.jdbc.NonRegisteringDriver"

If "classesToInitialize" is already set for you, simply add NonRegisteringDriver to the existing comma separated value.

and the answer is:

[Jun 8. 2013 21:33] Marco Asplund I did some testing using the JreMemoryLeakPreventionListener / classesToInitialize workaround (Tomcat 7.0.39 + MySQL Connector / J 5.1.25).

Before applying thread bypass dumps, several instances of AbandonedConnectionCleanupThread are listed after re-deploying webapp several times. After applying the workaround, there is only one instance of AbandonedConnectionCleanupThread.

I had to modify my application, although moving the MySQL driver from webapp to Tomcat lib. Otherwise, the class loader will not be able to load com.mysql.jdbc.NonRegisteringDriver when starting Tomcat.

I hope this helps everyone who is still struggling with this problem ...

Answer 8

See To prevent a memory leak, the JDBC driver was forcibly unregistered . Bill is responsible for registering all instances of the driver, as well as instances that may belong to other web applications. I extended Bill's answer by verifying that the driver instance belongs to the ClassLoader rule.

Here is the resulting code (in a separate method, because I have contextDestroyed other things):

 // See https://stackoverflow.com/questions/25699985/the-web-application-appears-to-have-started-a-thread-named-abandoned-connect // and // /questions/26009/to-prevent-a-memory-leak-the-jdbc-driver-has-been-forcibly-unregistered/190250#190250 private void avoidGarbageCollectionWarning() { ClassLoader cl = Thread.currentThread().getContextClassLoader(); Enumeration<Driver> drivers = DriverManager.getDrivers(); Driver d = null; while (drivers.hasMoreElements()) { try { d = drivers.nextElement(); if(d.getClass().getClassLoader() == cl) { DriverManager.deregisterDriver(d); logger.info(String.format("Driver %s deregistered", d)); } else { logger.info(String.format("Driver %s not deregistered because it might be in use elsewhere", d.toString())); } } catch (SQLException ex) { logger.warning(String.format("Error deregistering driver %s, exception: %s", d.toString(), ex.toString())); } } try { AbandonedConnectionCleanupThread.shutdown(); } catch (InterruptedException e) { logger.warning("SEVERE problem cleaning up: " + e.getMessage()); e.printStackTrace(); } } 

I wonder if calling AbandonedConnectionCleanupThread.shutdown() is safe. Can it interfere with other web applications? Hopefully not, because the AbandonedConnectionCleanupThread.run() method is not static, but the AbandonedConnectionCleanupThread.shutdown() method.

Answer 9

This seems to have been fixed in 5.1.41 . You can upgrade Connector / J to 5.1.41 or later. https://dev.mysql.com/doc/relnotes/connector-j/5.1/en/news-5-1-41.html

The implementation of AbandonedConnectionCleanupThread is now improved, so now there are four ways for developers to deal with the situation:

• When the default Tomcat configuration is used and the connector / J jar is placed in the local library directory, the new built-in application detector in Connector / J now detects the web application has stopped for 5 seconds and kills AbandonedConnectionCleanupThread. Any unnecessary warnings about the inability to stop the thread are also avoided. If the Connector / J jar is placed in the global library directory, the thread remains on until the JVM is unloaded.

• When the Tomcat context is configured with the clearReferencesStopThreads = "true" attribute, Tomcat is about to stop all spawned threads when the application stops if Connector / J is not shared with other web applications, in which case Connector / J is now protected from inappropriate stopping Tomcat ; a non-stoppable thread warning is still thrown in the Tomcat error log.

• When ServletContextListener is implemented in every web application that calls AbandonedConnectionCleanupThread.checkedShutdown () when the context is destroyed, now Connector / J skips this operation if the driver is potentially available to other applications. In this case, a warning is not displayed in the Tomcat error log that a thread that stops being locked is issued.

• When AbandonedConnectionCleanupThread.uncheckedShutdown () is called, AbandonedConnectionCleanupThread closes, even if Connector / J is shared with other applications. However, it may subsequently be impossible to restart the stream.

If you look at the source code, they called setDeamon (true) in the stream, so it will not block shutdown.

 Thread t = new Thread(r, "Abandoned connection cleanup thread"); t.setDaemon(true);