Java neo4j, REST and memory

Question

I deployed the application using the built-in java version of neo4j under the Jersey tomcat for the REST API. Measuring memory usage with jconsole, I noticed that every REST call adds 200 MB of memory (which, I think, because the entire graph is loaded into memory). Therefore, in just 5 calls, the server allocates 1 GB of memory, which is a lot! To clear the memory, I have to wait for the garbage collector (the threshold is set to 1 GB).

Is this normal behavior because I'm using the built-in java version of neo4j, or am I doing something terribly wrong? What should I do to free up memory when the API call ends?

Here is a sample code:

@GET @Produces(MediaType.APPLICATION_JSON + ";charset=utf-8") public Response getApi( @QueryParam("q") String query){ try{ // new neo instance here with EmbeddedGraphDatabase ... some code // stop neo }catch(Exception ex){ // stop neo } return response.ok("json data here").build(); } 

Thanks Daniele

-------- FULL CLASS CLASS ----------

  import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; import javax.ws.rs.QueryParam; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; import org.neo4j.graphdb.GraphDatabaseService; import org.neo4j.kernel.EmbeddedGraphDatabase; @Path("/API") public class API { @GET @Produces(MediaType.APPLICATION_JSON) public Response apiCall(@QueryParam("q") String query){ GraphDatabaseService graphDb; try{ // start neo graphDb = new EmbeddedGraphDatabase( "/var/neo4jdb/" ); this.registerShutdownHook( graphDb ); // API + NEO code here.. // stop neo graphDb.shutdown(); }catch(Exception ex){ // stop neo graphDb.shutdown(); } Response response = null; return response.ok("This is your query: "+query).build(); } /** * Server shutdown */ public void registerShutdownHook( final GraphDatabaseService graphDb ){ // Registers a shutdown hook for the Neo4j instance so that it // shuts down nicely when the VM exits (even if you "Ctrl-C" the // running example before it completed) Runtime.getRuntime() .addShutdownHook( new Thread() { @Override public void run() { graphDb.shutdown(); } } ); } } 

And then I call the REST service through the browser as follows http://localhost:8080/API?q=test

UPDATED WITH SINGLET

  import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; import javax.ws.rs.QueryParam; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; import org.neo4j.graphdb.GraphDatabaseService; import org.neo4j.kernel.EmbeddedGraphDatabase; @Path("/API") public class API { @GET @Produces(MediaType.APPLICATION_JSON) public Response apiCall(@QueryParam("q") String query){ GraphDatabaseService graphDb; try{ // start neo Neo4jSingleton neo4jInstance = new Neo4jSingleton(); GraphDatabaseService graphDb = null; graphDb = neo4jInstance.getInstance(DB_PATH); this.registerShutdownHook( graphDb ); // API + NEO code here.. // cypher query ExecutionEngine engine = new ExecutionEngine(graphDb); String queryString = "Cypher query code"; ExecutionResult result = engine.execute( queryString ); // fetch results here.. // never stop neo now with singleton }catch(Exception ex){ // stop neo graphDb.shutdown(); } Response response = null; return response.ok("This is your query: "+query).build(); } /** * Server shutdown */ public void registerShutdownHook( final GraphDatabaseService graphDb ){ // Registers a shutdown hook for the Neo4j instance so that it // shuts down nicely when the VM exits (even if you "Ctrl-C" the // running example before it completed) Runtime.getRuntime() .addShutdownHook( new Thread() { @Override public void run() { graphDb.shutdown(); } } ); } } public class Neo4jSingleton { private static GraphDatabaseService db; public Neo4jSingleton() { } /* * il metodo di restituire un'unica istanza * contenente il database neo4j */ public static GraphDatabaseService getInstance(String DB_PATH) { //Boolean isDbChange=verifyDbChange(); if (db == null /*|| isDbChange*/) { db = new EmbeddedGraphDatabase(DB_PATH); } return db; } } 

Answer 1

You do not have to create an instance of Neo4j for each request. Please create it only once, and then pass it, either hacked into a static field (since the resource instance is recreated for each request), or as on a Neo4j server with a Provider that @Context is @Context .

Regarding memory usage. Neo4j creates internal caches according to your needs in order to execute the same requests at the same time faster. Thus, this may affect some of the memory used.

Btw. how big is your chart and what are the typical operations you perform?

Answer 2

Java will not necessarily recover memory once it becomes GCed. You may find that you do not see significant GC action until you get close to the heap limit. Therefore, even if you put a limit on the heap of 10gb, you may find that memory jumps up again. This is not necessarily a problem.

However, to solve the Neo problem using N x Memory, you should consider sharing the embedded instance between all of your endpoints. Perhaps by moving it to a service class and having a common instance between endpoints. If you use Spring with Jersey, then this will be easy to do, since you can connect it through Spring.

Answer 3

If you are not using Spring or Guice, you can work around this using the static member and static initialization for your neo4J object. If a transaction breaker is used for this statics, another option is to use a static factory to get one instance of this neo4J object ...