Python Celery vs. Threading library to run asynchronous requests

Question

I am running a python method that parses a lot of data. Since this is an intensive time, I would like to run it asynchronously in a separate thread so that the user can access the website / interface.

Are threads using the "thread import stream" running if the user leaves the site or continues to work on the server?

What would be the advantages of using Celery compared to just using a thread module for something like that?

Answer 1

Python threads are not true OS threads. See Green threads and thread in python .

And, the Python interpreter is decidedly single-threaded due to (c) the famous global lock aka (GIL) interpreter. Thus, threads in Python only provide parallelism when computing and IO can be done simultaneously. Computing related tasks will be of little use for streaming in the Python threading model, at least under CPython 2 or 3.

On the other hand, these restrictions do not apply to multiprocessing, which you will do with the queuing system, for example, celery. You can run multiple working instances of Python that can run simultaneously on multi-core machines or on multiple machines.

If I understand your scenario - the interaction on the website begins with a long work - the queue is almost certainly the way to go. You get the true value of parallelism and an easy way to transfer processing to other machines.