Python web application with background workflow

Question

Requirements:

• Define a “job” that has a start time t, batch size s, interval i, and a list to be processed
• Starting at time t, capture the following items from the list every i second and process it
• Work can be paused and resumed (the user should be able to say the task to stop capturing new list items for processing).

The checkbox will be used for the web application. Obviously, I need some kind of background process / thread that will periodically execute the processing code.

Since the state will be stored in the database, the easiest approach I can imagine is to define a cronjob that will periodically execute a python script that checks active jobs and does the processing.

Any suggestion on how to do this using python only?

• Run another python process that will periodically check and execute active jobs?
• Create workflow from Flask?
• ...?

Answer 1

I would highly recommend using a queuing mechanism such as Redis or RabbitMQ. The flask will act as a producer, and your "worker" will consume and process.

Setting up any of these tools is much less complicated than you might expect.

sudo apt-get install redis-server sudo apt-get install python-pip sudo pip install redis 

Your flask application acts as a manufacturer

 >>> from redis import Redis >>> r = Redis() >>> r.lpush('task_queue', 'task1') 1L 

And your “worker” consumes and processes asynchronously:

 >>> r.rpop('task_queue') 'task1'