Web application: hold a large object between requests

Question

I am working on a genome search web application. This application uses this suffix library through Cython bindings. Objects of this type are large (from hundreds of MB to ~ 10 GB) and take as long as it takes to load from disk, as is required to process them in response to a page request. I am looking for a way to load several of these objects once at server load, and then use them for all page requests.

I tried using the remote manager / client setup using the multiprocessing module modeled after this demo , but it fails when the client connects to an error message that says the object is not being picked up.

Answer 1

I would suggest writing a small jar (or even raw WSGI ... But it's probably easier to use Flask, as it will be easier to get up and running quickly) an application that loads the genome database, then provides a simple API. Something like that:

app = Flask(__name__) database = load_database() @app.route('/get_genomes') def get_genomes(): return database.all_genomes() app.run(debug=True) 

Or, you know, something more reasonable.

Also, if you need to process multiple requests at a time (I believe that app.run will only process one at a time), start streaming ... And if it's too slow, you can os.fork() after the database loads and launches several request handlers from there (thus, they will all share the same database in memory).