Troubleshoot memory leaks with hot redeployment and Oracle database

Question

I have a SpringRoo-based application that runs in a production environment, which causes severe memory leaks in permg after some hot redeployments.

To “find and fix” a leak and reduce variables during analysis, I created a simple, simplified application using roo, and I get the same behavior. A project (created using Spring Roo (1.2.3.RELEASE)) simply saves an object called "Man" with a string field called "name".

I am deploying the war on Tomcat 7.0.39 using Oracle 11.2.0.2 as the database. After each redeployment, I get this message in catalina.out

INFO: Undeploying context [/ojdbc-0.1.0.BUILD-SNAPSHOT] mag 06, 2013 10:50:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [/ojdbc-0.1.0.BUILD-SNAPSHOT] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. 

After two hot redeployments, I get a permg error:

 mag 06, 2013 10:51:08 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /Applications/apache-tomcat-7.0.39/webapps/ojdbc- 0.1.0.BUILD-SNAPSHOT.war Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" Exception in thread "RMI TCP Connection(idle)" mag 06, 2013 10:51:17 AM ServerCommunicatorAdmin reqIncoming WARNING: The server has decided to close this client connection. java.lang.OutOfMemoryError: PermGen space Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space 

I tried to analyze this using the VisualVm / EclipseMemory Analyzer, and this is what I still get.

The fact is that I do not observe this behavior with another database (for example, PostgreSQL or Hypersonic). Is there something Oracle related that is causing a leak?

Here is the zip archive containing the roo script generator, anche .hprof dump file.

Answer 1

Try moving the Oracle JDBC driver to the Tomcat lib directory instead of being inside the lib application folder. It looks like OracleDiagnosabilityMBean is getting a handle on Catalina.

Edit: since you don't have control over Tomcat, try wrapping it where Oracle classes are loading like this (except for replacing the AppContext for your Oracle init):

http://cdivilly.wordpress.com/2012/04/23/permgen-memory-leak/

 //somewhere in application startup, eg the ServletContextListener try { final ClassLoader active = Thread.currentThread().getContextClassLoader(); try { //Find the root classloader ClassLoader root = active; while (root.getParent() != null) { root = root.getParent(); } //Temporarily make the root class loader the active class loader Thread.currentThread().setContextClassLoader(root); //Force the AppContext singleton to be created and initialized sun.awt.AppContext.getAppContext(); } finally { //restore the class loader Thread.currentThread().setContextClassLoader(active); } } catch ( Throwable t) { //Carry on if we get an error LOG.warning("Failed to address PermGen leak"); } 

Answer 2

There is an internal Oracle error report

Oracle Support Bug 18707079 (JDBC THIN DRIVER LEAKS THREADS AND REFERENCES ON WEBAPP SHUTDOWN): https://support.oracle.com/epmos/faces/BugDisplay?id=18707079

which describes your problem:

 When the webapp is shutdown, customer experience the following problems: 1) The OracleDiagnosabilityMBean is not de-registered 2) OracleTimeoutPollingThread does not stop 3) The following thread does not stop: oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser 

It is installed in Fixed in Product Version 12.2 . Hope it comes out soon!

Answer 3

Try the following:

Register ServletContextListener in web xml

 <listener> <listener-class>my.package.MyShutdownServletContextListener</listener-class> </listener> 

A source:

 public class MyShutdownServletContextListener implements ServletContextListener { @Override public void contextInitialized(ServletContextEvent event) { final ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader(); try { Thread.currentThread().setContextClassLoader(ClassLoader.getSystemClassLoader()); java.sql.DriverManager.getDrivers(); Class.forName("oracle.jdbc.driver.OracleTimeoutThreadPerVM"); } catch (ClassNotFoundException e) { /* noop */ } finally { Thread.currentThread().setContextClassLoader(contextClassLoader); } } @Override public void contextDestroyed(ServletContextEvent event) {} }