Gunicorn, Django, Gevent: spawned threads block

Question

We recently switched to Gunicorn using gevent .

On our website, we have several tasks that take time. More than 30 seconds.

Preamble

We already did the whole celery thing, but these tasks are performed so rarely that it is simply impossible to complete celery and redis all the time. We just don't want that. We also do not want to start celery and process on demand. We want to get rid of him. (I apologize for this, but I want to prevent answers that sound like this: “Why don’t you use celery, it's great!”)

Tasks we want to perform asynchronously

I am talking about tasks that run 3,000 SQL queries (inserts) that must be executed one after another. This is not done too often. We limited ourselves to simultaneously performing only two of these tasks. They should take like 2-3 minutes .

An approach

Now, what we do now, we use gevent work and gevent.spawntask and return the answer.

Problem

I found that the generated threads are actually blocking. As soon as the response is returned, the task starts and no other requests are processed until the task stops working. The task will be killed after the 30s, gunner-shooter timeout. To prevent this, I use time.sleep()after every other SQL query, so the server gets the opportunity to respond to queries, but I do not feel that this is the point.

, gevent. dev 1 gevent worker. ( ). , 2 , , .

TL;DR

• gevent 2- ( )
• gunicorn gevent , gevent.spawn
• ?

!

Answer 1

synchronous () multiprocessing. .

, memcached, , .

Answer 2

- fork . Gevent, - . , ( ) , .

: . , return Gunicorn, .

. , exit. , :

    if os.fork():
        return JsonResponse({}) # async parent: return HTTP 200
    # child: do stuff, exit quietly
    ret = do_tag_notify(
        event, emails=emails, photo_names=photo_names,
        )
    logging.info('do_tag_notify/async result={0}'.format(ret))
    os._exit(0)             # pylint: disable=W0212
    logging.error("async child didn't _exit correctly") # never happens

. , , ! . .

fork - - !

Answer 3

, .

?

- . SQL- - - . SQL/ORM, gevent, - . .

join , .