How to develop a process in python / django?

Question

This is a more general Python question, however, in the context of django.

At the moment, I have this view in django that should handle a lot of data. This usually requires a server (nginx with django running in a proxy server) for a couple of minutes. Sometimes the server is interrupted. I do not want to increase the latency in nginx. I understand that if I can develop a process in python in a django view so that the processed (child) process executes all the data that is crunchy regardless of the django view, then the view could immediately return the request to the user (so there will never be a timeout), and the child process will continue to run in the background, ending all calculations.

So here is the question:

How can I convert an independent process in python (and, if possible, for python code in the same file)? And if possible, how can I give it the priority level of the unix process?

I looked at some ways to fork a process in python, and it seems that there are several options. Which one is most suitable for this scenario?

Thanks.

Answer 1

The "best practice" answer is to use a queue manager, usually RabbitMQ or any backend processed by a Django celery.

However, there are several lighter options that spawn a new stream. what is usually lacking in these parameters is some way to track progress or control the number of threads.

check out Django-utils to see if this is enough. if not, go for Celery.

Answer 2

If you really want to unlock and set priority, you can use os.fork and os.nice , but I think the multiprocessing or Celery module will be more applicable here.