The python process takes time to run in a django project running on nginx and uwsgi

Question

I am starting a process using the python multiprocessing module. This process is invoked by an email request sent to the django project. When I use the development server (python manage.py runningerver), the post request does not take time to start the process and ends immediately.

I launched a production project using nginx and uwsgi.

Now, when I submit the same send request, it takes about 5-7 minutes to complete this request. This only happens with those mail requests in which I start the process. Other mail requests work fine.

What could be causing this delay? And how can I solve this?

Answer 1

I figured out a workaround (I don't know if it will qualify as an answer). I wrote the background process as a job in the database and used cronjob to check if I have any work and if any cron starts the background process for this job and quits.

Cron will start every minute so there is not much delay. This helped improve performance, as it helped me complete heavy tasks like this, to run separately from the main application.

Answer 2

Basically, background processing should start outside the WSGI application module.

WSGI webapp python , . , WSGI , .

, WSGI , WSGI. - . WSGI webapp, .

your_app/webapp.py:

from . import bg_queue
def post():
    # Webapp POST code here
    bg_queue.add(<task data>)

your_app/processor.py:

from multiprocessing import Process

class Consumer(Process):
    def __init__(self, input_q):
        self.input_q = input_q

    def run(self):
        while True:
            task_data = input_q.get()
            <process data>

your_app/__init__.py:

from .processor import Consumer
bg_queue = Queue()

consumer = Consumer(bg_queue)
consumer.daemon = True
consumer.start()