I analyzed the leak of the class loader in the last few days in a large application, and I solved the problem.
My application uses SolrJ, which will be initialized with @Bean -Method:
@Bean(destroyMethod = "close") public SolrClient solrClient() { return new HttpSolrClient(SOLR_URL); }
SolrJ ( org.apache.solr:solr-solrj:5.4.1 ) uses the Apache HttpClient ( org.apache.httpcomponents:httpclient:4.4.1 ). HttpClient initializes the SSL context using regular java classes such as javax.net.ssl.SSLSocketFactory . Thus, java loads trustManager and parses all the trusted certificates. If there is an error, the certificate (instance of sun.security.x509.X509CertImpl ) is saved in the list and receives an enriched exception. This exception is swallowed and my application is not aware.
As far as I understand, the SSL context is in the System / Root Classloader, my application is in a dedicated WebappClassLoader , and this is a problem, because now there is an IOException inside the SSL context that contains links in stacktrace, backtrace, etc. classes in my application.
But now I don’t know where it came from. Is it a SolrJ client, Apache HttpClient, Java itself (JVM) or is it my application?
I made a small application to reproduce the problem, which you can find here: https://github.com/CptS/solrj-classloader-leak This also contains a workaround (a disconnect hook that removes links that cause the classloader to leak).
If you turn off the shutdown hook (for example, by commenting on it) and run a clean Tomcat (see “Playback Environment” below), you can play it by following these steps:
- unleash the demo project war (A)
- reload it (B)
- reload it again (C)
- Trigger GC (D)
- Undeploy
- Trigger GC (E)
- See metaspace not completely cleared (F)
I created a bunch of heaps, and the shortest path to the GC is as follows:
It was the same as in my big application. The mentioned workaround (a bit inspired by https://github.com/mjiderhamn/classloader-leak-prevention , but this unfortunately does not solve my problem), using reflection for these unparseableExtensions and removing the exception stored in the why field as follows : SSLContextImpl.DefaultSSLContext#defaultImpl → SSLContextImpl#trustManager → X509TrustManager#trustedCerts → X509CertImpl#info → X509CertInfo#extensions → CertificateExtensions#unparseableExtensions → UnparseableExtension#why
Having done this, I have stacktrace exceptions if it helps someone:
java.io.IOException: No data available in passed DER encoded value. at sun.security.x509.GeneralNames.<init>(GeneralNames.java:61) at sun.security.x509.IssuerAlternativeNameExtension.<init>(IssuerAlternativeNameExtension.java:136) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at sun.security.x509.CertificateExtensions.parseExtension(CertificateExtensions.java:113) at sun.security.x509.CertificateExtensions.init(CertificateExtensions.java:88) at sun.security.x509.CertificateExtensions.<init>(CertificateExtensions.java:78) at sun.security.x509.X509CertInfo.parse(X509CertInfo.java:702) at sun.security.x509.X509CertInfo.<init>(X509CertInfo.java:167) at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1804) at sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:195) at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:100) at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:755) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:226) at sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultTrustManager(SSLContextImpl.java:767) at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:733) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.security.Provider$Service.newInstance(Provider.java:1595) at sun.security.jca.GetInstance.getInstance(GetInstance.java:236) at sun.security.jca.GetInstance.getInstance(GetInstance.java:164) at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96) at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122) at org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSocketFactory.java:190) at org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(SchemeRegistryFactory.java:85) at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:121) at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:484) at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:234) at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:40) at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:149) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:125) at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:189) at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:162) at de.test.spring.SolrJConfig.solrClient(SolrJConfig.java:20) at de.test.spring.SolrJConfig$$EnhancerBySpringCGLIB$$dbd4362f.CGLIB$solrClient$0(<generated>) at de.test.spring.SolrJConfig$$EnhancerBySpringCGLIB$$dbd4362f$$FastClassBySpringCGLIB$$8e7566a6.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:309) at de.test.spring.SolrJConfig$$EnhancerBySpringCGLIB$$dbd4362f.solrClient(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1119) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1014) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) at de.test.WicketApplication.init(WicketApplication.java:32) at org.apache.wicket.Application.initApplication(Application.java:950) at org.apache.wicket.protocol.http.WicketFilter.init(WicketFilter.java:429) at org.apache.wicket.protocol.http.WicketFilter.init(WicketFilter.java:353) at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:279) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:260) at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:105) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4640) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5247) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:724) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:919) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1703) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
My solution to the problem is currently solving the problem, but of course this is just a workaround.
I want to know, and maybe someone can answer one or more of my questions:
- Is this a “bug” in SolrJ, HttpClient, Java, or my application?
- If this is my application, what am I doing wrong?
- If this is not my application, is this a known issue? I can not find any information about this. (Where) should I create an error ticket?
- Why is there an "invalid" certificate? (By the way: Maybe the leak will also be resolved if I delete this certificate from the trust repository ... I have not tested it, but I think that an invalid or damaged certificate should never cause the class loader to leak ...)
- Does anyone have more info on this? I cannot believe that I am the only one who detects this behavior (with the exception of my application ... see my question 2).
Last but not least, my playback environment:
- Tomcat Version: Apache Tomcat / 8.0.14 (Debian)
- JVM Version: 1.8.0_91-b14
- JVM Vendor: Oracle Corporation
- OS Name: Linux
- OS Version: 3.16.0-4-amd64
- Architecture: amd64