How to use Neo4j built-in for Python (threads) in Flask microframe?

Question

I follow the Flask tutorial (Flaskr) to experiment using Neo4j-embedded for Python. It is in the virtual space. Here is my "main" application code:

import os import jpype from neo4j import GraphDatabase from flask import Flask, request, session, g, redirect, url_for, abort, render_template, flash app = Flask(__name__) app.config.from_pyfile(os.environ['APP_SETTINGS']) def connectDB(): return GraphDatabase(app.config['DATABASE']) def initDB(): db = connectDB() with db.transaction: users = db.node() roles = db.node() db.reference_node.USERS(users) db.reference_node.ROLES(roles) userIndex = db.node.indexes.create('users') user = db.node(name=app.config['ADMIN']) user.INSTANCE_OF(users) userIndex['name'][app.config['ADMIN']] = user role = db.node(type='superadmin') role.INSTANCE_OF(roles) role.ASSIGN_TO(user) db.shutdown() print "Database initialized." def testDB(): db = connectDB() with db.transaction: userIndex = db.node.indexes.get('users') user = userIndex['name'][app.config['ADMIN']].single username = user['name'] db.shutdown() print "Admin username is '%s'. Database exists." % username @app.before_request def before_request(): jpype.attachThreadToJVM() g.db = connectDB() @app.teardown_request def teardown_request(exception): g.db.shutdown() @app.route('/') def index(): with g.db.transaction: userIndex = g.db.node.indexes.get('users') user = userIndex['name'][app.config['ADMIN']].single username = user['name'] fields = dict(username=username) return render_template('index.html', fields=fields) if os.path.exists(app.config['DATABASE']) == False: initDB() else: testDB() 

initDB () and testDB () work fine - without Gremlin, PyLucene, etc. - just jpype and neo4j-embedded. Initially, the JVM failed and the application crashed when I request index (). I looked through the network to find out that I need to add the line "jpype.attachThreadToJVM ()" to before_request () to solve this problem with python by typing JVM and the application does not exit. However, this immediately leads to another problem:

 Traceback (most recent call last): File "/ht/dev/envFlask/lib/python2.7/site-packages/flask/app.py", line 1518, in __call__ return self.wsgi_app(environ, start_response) File "/ht/dev/envFlask/lib/python2.7/site-packages/flask/app.py", line 1506, in wsgi_app response = self.make_response(self.handle_exception(e)) File "/ht/dev/envFlask/lib/python2.7/site-packages/flask/app.py", line 1504, in wsgi_app response = self.full_dispatch_request() File "/ht/dev/envFlask/lib/python2.7/site-packages/flask/app.py", line 1264, in full_dispatch_request rv = self.handle_user_exception(e) File "/ht/dev/envFlask/lib/python2.7/site-packages/flask/app.py", line 1262, in full_dispatch_request rv = self.dispatch_request() File "/ht/dev/envFlask/lib/python2.7/site-packages/flask/app.py", line 1248, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/ht/dev/apps/evobox/evobox/__init__.py", line 68, in index userIndex = g.db.node.indexes.get('users') File "/ht/dev/envFlask/lib/python2.7/site-packages/neo4j/index.py", line 36, in get return self._index.forNodes(name) java.lang.RuntimeExceptionPyRaisable: java.lang.IllegalArgumentException: No index provider 'lucene' found. Maybe the intended provider (or one more of its dependencies) aren't on the classpath or it failed to load. 

A Google search of the entire last line will not go anywhere. Just searching for "java.lang.IllegalArgumentException: the index provider" lucene "was not found. Do not give anything in the python context.

Neo4j messages.log seems to indicate that the database has been opened 3 times (initDB (), testDB () and index ()). The class path is the same for each instance:

 Class Path: /ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/neo4j-jmx-1.5.M02.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/neo4j-lucene-index-1.5.M02.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/neo4j-graph-matching-1.5.M02.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/neo4j-1.5.M02.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/neo4j-kernel-1.5.M02.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/geronimo-jta_1.1_spec-1.1.1.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/lucene-core-3.1.0.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/neo4j-graph-algo-1.5.M02.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/neo4j-udc-1.5.M02.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/neo4j-cypher-1.5.M02.jar:/ht/dev/envFlask/local/lib/python2.7/site-packages/neo4j/javalib/scala-library-2.9.0-1.jar 

I also modified index () for connectDB and attachThreadToJVM directly, like initDB () and testDB (), without using the global 'g' - this led to the same error.

What am I possibly missing / missing to embed neo4j and jpype in a threaded request, and not just in the "main" application?

Note. I know RESTful Web Service solutions with py2neo or Rexster / Bulbs, but now I want to avoid this.

EDIT: using JPype-0.5.4.2, Neo4j-embedded-1.5.b2, Java-6-openjdk

Answer 1

One problem with this pattern is that you run the risk of running multiple databases listed in the same place, which will lead to problems. What you want is one database instance that follows the full life cycle of your application.

Why doesn't he find the lucene provider? This is a harder question. The index provider is loaded using a java service, which means that JPype should not affect it. As long as the JVM starts up fine and the juc implementation of lucene-index is on the class path, it should work.

Perhaps this is due to stream processing, I'm currently writing a patch to automatically handle the calls to "attachThreadToJVM ()". I will add a test case to make sure that reading indexes from a separate thread works just as expected.

In this mailing list thread, the thread work is updated:

http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-Python-embedding-problems-with-shutdown-and-threads-td3476163.html

Answer 2

Tried to replicate the error using a clean install of neo4j, flask and jpipe using the standard java default path. Failed to create this log. But I got an error

 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "teststack.py", line 27, in initDB userIndex = db.node.indexes.create('users') File "/usr/local/lib/python2.7/dist-packages/neo4j/index.py", line 32, in create return self._index.forNodes(name, to_java(config)) jpype._jexception.RuntimeExceptionPyRaisable: java.lang.IllegalArgumentException: Supplied index configuration: {} doesn't match stored config in a valid way: {provider=lucene, type=exact} for 'users' 

when using initDB

with

 userIndex = db.node.indexes.create(users)