I am currently working on a webapp that uses Neo4j. Our application must be deployed in a Tomcat environment (client requirement). We decided to implement Neo4j, because in this way we can use Neo4j provided by the Java API, it is easier to deploy, and we get improved performance. However, we also need access to the REST API, because we have a one-page webapp written in Angular that currently uses this interface. However, the built-in Neo4j database does not expose REST api. The artifact of the Neo4j server contains code that can load the berth server with an integrated graphical database. Therefore, our Tomcat webapp launches the Jetty server. We can access the graph using the webapp deployed in Tomcat on one port (8080) and the Neo4j REST interface and the Neo4j browser on the other port (7474). Although this is a bit strange, it works just fine, unless we are trying to stop our webapp (for example, to redistribute it). When we close our webapp, we get these errors from tomcat:
SEVERE: The web application [/chainmonitor] appears to have started a thread named [GC-Monitor] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [RRD4J Sync Pool [Thread-1]] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [Statistics Gatherer[primitives]] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [pool-1-thread-2] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [DateCache] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-31-selector-3] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-32-selector-0] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-33-selector-1] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-34-selector-2] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-35-selector-4] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-36-selector-5] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [ qtp835579386-37-acceptor-0-ServerConnector@1425c689 {HTTP/1.1}{localhost:7474}] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [ qtp835579386-38-acceptor-1-ServerConnector@1425c689 {HTTP/1.1}{localhost:7474}] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [HashSessionScavenger-0] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-54] but has failed to stop it. This is very likely to create a memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [org.eclipse.jetty.http.HttpFields$1] (value [ org.eclipse.jetty.http.HttpFields$1@4a3d3a5e ]) and a value of type [org.eclipse.jetty.http.HttpFields.DateGenerator] (value [ org.eclipse.jetty.http.HttpFields$DateGenerator@24f0137b ]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$ $anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [ scala.util.DynamicVariable$$anon$1@281a099d ]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. jun 10, 2014 6:43:27 PM org.apache.catalina.startup.HostConfig undeploy INFO: Undeploying context [/chainmonitor] jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar deleteDir SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor\WEB-INF\lib] could not be completely deleted. The presence of the remaining files may cause problems jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar deleteDir SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor\WEB-INF] could not be completely deleted. The presence of the remaining files may cause problems jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar deleteDir SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor] could not be completely deleted. The presence of the remaining files may cause problems jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar delete SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor] could not be completely deleted. The presence of the remaining files may cause problems
Our pom.xml is as follows:
<dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>2.0.3</version> </dependency> <dependency> <groupId>org.neo4j.app</groupId> <artifactId>neo4j-server</artifactId> <version>2.0.3</version> </dependency>
Here's how we launch the embedded database and launch the berth server:
graphDb = (EmbeddedGraphDatabase) new GraphDatabaseFactory(). newEmbeddedDatabaseBuilder(dataDir).loadPropertiesFromURL(Neo4jPropertiesUrl).newGraphDatabase(); bootstrapper = new WrappingNeoServerBootstrapper(graphDb);
Neo4j-server.properties is currently empty. I have not tried many different configurations, but this does not seem to be a problem.
Here's how we stop the embedded database and the berth server:
@Override public void contextDestroyed(ServletContextEvent event) { graphDb.shutdown(); bootstrapper.stop(); }
This is apparently caused by Tomcat's memory leak protection ( http://wiki.apache.org/tomcat/MemoryLeakProtection ).
As a result, our webapp can only be redistributed by completely destroying the Tomcat process (kill -9), which is very undesirable.
We tried a bunch of things:
- Do not disable the built-in grapbDb. There is no effect.
- does not explicitly disable the bootloader. There is no effect. (the boot device has its own disconnect hook)
- first turn off the bootloader, and then the built-in database.
- Stream Simulation: https://github.com/Neo4j/Neo4j/issues/1070 . The code starts and destroys 59 threads of local values, but without a visible effect. Interestingly, when I run this in a loop for a minute during shutdown, it keeps saying that it destroyed 6 values. For me, this indicates that this process does not work at all, because Neo4j takes some time to shut down.
- delay after closing. The reason we tried this is because after tomcat received a shutdown signal, we need to wait a few seconds before actually killing the process, otherwise we will get an error the next time that our graph data was corrupted (with message that was not properly closed). A delay of 5+ seconds also seems to have an effect, because after that we get only 6 serious errors instead of 16. This is also the reason why I think the memorial process does not work at all.
- We tried as an alternative to add our built-in Neo4j to the cluster, thus being able to use both the REST api and the built-in Neo4j. However, after about four hours, trying to get this to work, we gave up. During the launch of the Neo4j embedded database, our web application simply froze. After some debugging, this is very much like a dead end in Neo4j code. We tried to set all HA timeout configurations to slightly lower values, but without effect.
The problem is caused by the Neo4j pier server. Without this, this problem does not occur. However, we need a REST API. Do you know how we can fix this? Or do you have an alternative way to use the embedded database and REST API in Tomcat?
We use:
- Java: 1.7.0_60
- Tomcat 7.0.54
- Neo4j: 2.0.3 (we tried 2.1.1, but got exactly the same result)
- The same thing is observed in Windows 7 and Linux Redhat