Gracefully stopping logback in a container environment

Question

Today I get PermGen OutOfMemory error.

Analysis showed that the closest GC Root for WebappClassLoader is the log stream:

 this - value: org.apache.catalina.loader.WebappClassLoader #4 <- contextClassLoader (thread object) - class: java.lang.Thread, value: org.apache.catalina.loader.WebappClassLoader #4 

which the:

 java.lang.Thread#11 - logback-1 

Dump the thread from the dump heap for this thread:

 "logback-1" daemon prio=5 tid=34 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) Local Variable: java.util.concurrent.SynchronousQueue$TransferStack$SNode#1 Local Variable: java.util.concurrent.SynchronousQueue$TransferStack#6 at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) Local Variable: java.util.concurrent.SynchronousQueue#6 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) Local Variable: java.util.concurrent.ThreadPoolExecutor#34 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#11 at java.lang.Thread.run(Thread.java:745) 

I am using Tomcat 8 with the hot reloadable="true" function reloadable="true" and the external CLASSPATH via PreResources :

 <Context docBase="/home/user/devel/app/src/main/webapp" reloadable="true"> <Resources> <!-- To override application.properties and logback.xml --> <PreResources className="org.apache.catalina.webresources.DirResourceSet" base="/home/user/devel/app/.config" internalPath="/" webAppMount="/WEB-INF/classes" /> </Resources> </Context> 

and logback.xml with scan="true" :

 <configuration debug="false" scan="true" scanPeriod="5 seconds"> <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> ... 

After saving the changes to the /home/user/devel/app/.config/logback.xml Tomcat 8 receipt notification (I'm not sure which API was used to monitor the changes in fs), and the application /home/user/devel/app/.config/logback.xml was started. This will happen before PermGen OutOfMemory .

How can I gracefully stop Logback in a container environment?

How to stop the stream "logback-1" ?

I found some related discussions, but can't figure out what to do with this information:

• http://logback.10977.n7.nabble.com/How-to-stop-all-appenders-td3023.html
• Do I need to disable events when closing using the log?
• The right way to stop a custom asynchronous appbackback
• Stop registration system for clean termination

UPDATE I play with a bunch of heaps in visualvm . Under the link level, go from the bad logback-1 threads:

 lvl1 = flatten(filter(referees(heap.findObject(0xf4c77610)), "!/WebappClassLoader/(classof(it).name)")) lvl2 = flatten(map(lvl1, "referees(it)")) lvl3 = flatten(map(lvl2, "referees(it)")) 

he refers to

 ch.qos.logback.core.util.ExecutorServiceUtil$1 

By grepping in the log sources for ExecutorServiceUtil I found an entry in the change list :

All topics opened by ch.qos.logback.core.util.ExecutorServiceUtil # THREAD_FACTORY are now the daemons that fix the application, freeze when they shut down when LoggerContext # stop () is not called (LOGBACK-929). Note that daemon threads are abruptly interrupted by the JVM, which can cause unwanted results, such as corrupted files written by FileAppender. It is still highly recommended for applying gracefully closing applications to call LoggerContext # stop () (for example, at shutdown).

Is it right that in the stream of container daemons are dangerous and lead to memory leaks?

Answer 1

I don’t quite understand what to do. I am currently removing the jul-to-slf4j from the pom.xml project and this line:

 <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator"/> 

from logback.xml . Even in this line application there is no "logback-1" stream.

As official documents suggest I will register:

 public class ShutdownCleanupListener implements ServletContextListener { @Override public void contextInitialized(ServletContextEvent sce) { } @Override public void contextDestroyed(ServletContextEvent sce) { if (LoggerFactory.getILoggerFactory() instanceof LoggerContext) { ((LoggerContext) LoggerFactory.getILoggerFactory() ).stop(); } } } 

in web.xml :

 <listener> <listener-class>com.app.servlet.ShutdownCleanupListener</listener-class> </listener> 

To remove a direct dependency on:

 import ch.qos.logback.classic.LoggerContext; 

You can use reflection.

Not sure I'm right. I will see if get PermGen OutOfMemory error due to Logback.

UPDATE After I discovered the link to the link from the ExecutorServiceUtil class, I checked the log sources and found that this class creates streams with names such as bad above:

  thread.setName("logback-" + threadNumber.getAndIncrement()); 

This class is used only in ch.qos.logback.core.ContextBase and the stream inside:

 public void stop() { // We don't check "started" here, because the executor service uses // lazy initialization, rather than being created in the start method stopExecutorService(); started = false; } 

Please note that LoggerContext is a subclass of ContextBase , so the above solution really fixes my problem .

Answer 2

Is it right that in the stream of container daemons are dangerous and lead to memory leaks?

The daemon thread statement refers to the case of a standalone application (as in the error message ), where the JVM is supposed to shut down the application. The daemon thread prevents the JVM from stopping.

In the context of JavaEE, where the application server JVM remains unchanged compared to potentially multiple life cycles from multiple applications, daemon vs non-daemon does not affect the fact that current threads are the root of the GC .

Answer 3

According to the Spring Boot logback example project, you should close the context to clear the logging system: https://github.com/spring-projects/spring-boot/commit/10402a651f1ee51704b58985c7ef33619df2c110

Example:

 public static void main(String[] args) throws Exception { SpringApplication.run(SampleLogbackApplication.class, args).close(); }