Strange error while starting threads inside django application

Question

I am trying to do simple network ping from a django process. Django is deployed via apache mod_wsgi. But the code only works during the first run, returning the following error on subsequent runs.

Traceback: File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response 111. response = callback(request, *callback_args, **callback_kwargs) File "/var/www/propingui/ping/views.py" in ping 32. p = Pool(len(fls)) File "/usr/lib/python2.7/multiprocessing/pool.py" in __init__ 674. Pool.__init__(self, processes, initializer, initargs) File "/usr/lib/python2.7/multiprocessing/pool.py" in __init__ 134. self._repopulate_pool() File "/usr/lib/python2.7/multiprocessing/pool.py" in _repopulate_pool 197. w.start() File "/usr/lib/python2.7/multiprocessing/dummy/__init__.py" in start 73. self._parent._children[self] = None Exception Type: AttributeError at / Exception Value: '_DummyThread' object has no attribute '_children' 

Code below:

 from multiprocessing.pool import ThreadPool as Pool ... def _ping((host, firing_location)): pinger = Pyro4.Proxy("PYRONAME:" + firing_location) return pinger.ping(host) def ping(request): if request.method == 'POST': form = PingForm(request.POST) if form.is_valid(): host = form.cleaned_data['host'] fls = ['g1','a1'] p = Pool(len(fls)) noanswer = False try: jobs = p.map(_ping, zip([host]*len(fls), fls) ) except: jobs = [] ... return ... 

I tried to execute a Google error, but didn’t find anything, and I don’t understand the exact source of the problem. Interestingly, if I change ThreadPool to a multiprocessor pool, everything will work fine. I think this is somehow caused by problems with spawning threads inside django.

Answer 1

To use threading, I had the same problem, eventually refused and used the streaming module instead of ThreadPool , and now everything works fine, so I think that the problem should be with ThreadPool not Python threads at all.

You can implement your own ThreadPool or reuse what others have done (like this or this )

Answer 2

This is most likely a well-known Python error that occurs when using ThreadPool from thread.Thread thread.

There is no good solution for this problem, just workarounds, such as calling the following in the thread that you want to use ThreadPool:

 if not hasattr(threading.current_thread(), "_children"): threading.current_thread()._children = weakref.WeakKeyDictionary() 

Answer 3

I think this is due to mod_wsgi. It is not recommended to create a thread or forks inside a wsgi application, because this can interfere with how the web server generates workers for the application.

Perhaps you can create a pyro service that sends your messages to the whole server ...