Twisted thread with subprocess.

Question

I am trying to implement a service with Twisted, which is close enough to the finger tutorial found here: http://twistedmatrix.com/documents/current/core/howto/tutorial/intro.html p>

I have a basic.LineListener waiting for a command and then executing it, then I have a client connecting and issuing commands. The problem is that the team sometimes needs to do something else, and I use the python subprocess module for this. These are not just communication calls () that hang, which is a normal subprocess problem, and I know how to get past it. This is what the .Popen subprocess calls hang.

Here's the twisted server code:

from twisted.application import internet, service
from twisted.internet import protocol, reactor, defer, threads
from twisted.protocols import basic
import sys
import time
import subprocess

class MyProtocol(basic.LineReceiver):
    def lineReceived(self, line):
        self.go()
    def go(self):
        def writeResponse(message):
            self.transport.write(message + '\r\n')
            self.transport.loseConnection()
        threads.deferToThread(self.factory.action).addCallback(writeResponse)
    def connectionMade(self):
        self.lines = []

class ActionService(service.Service):
    def __init__(self, **kwargs):
        pass
        #self.users = kwargs

def action(self):
    print "launching subprocess"
    sys.stdout.flush()
    p = subprocess.Popen(["ls"], stderr=subprocess.PIPE, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
    print "launched subprocess, trying to communicate..."
    sys.stdout.flush()
    p.communicate()
    print "returning"
    sys.stdout.flush()
    return "%032d" % (0)

    def getActionFactory(self):
        f = protocol.ServerFactory()
        f.protocol = MyProtocol
        f.action = self.action
        return f

reactor.suggestThreadPoolSize(300)
application = service.Application('Action', uid=0, gid=0)
f = ActionService()
serviceCollection = service.IServiceCollection(application)
internet.TCPServer(31337,f.getActionFactory()
                   ).setServiceParent(serviceCollection)

... and here is some client code:

#!/usr/bin/python
import time
import threading
import socket

def connectAction(host):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((host, 31337))
    s.send("asdf\r\n")
    resp = s.recv(32)
    s.close()
    return resp

class sscceThread(threading.Thread):
    def __init__(self, host):
        self.host = host
        threading.Thread.__init__(self)
    def run(self):
        connectAction(self.host)

def main():
    threads = []
    for i in range(0, 1000):
        for j in range(0,5):
            t = sscceThread("localhost")
            t.start()
            threads.append(t)
        for t in threads:
            t.join()
        print i
        time.sleep(1)
    #    print i

if __name__ == "__main__":
    main()

Start the service by running:

twistd -y sscce_twisted_service.py -l twistdLog; tail -f twistdLog

And start the client by running:

./sscce_twisted_client.py

, ( , 10), . 1- , , , , - :

2009-12-22 11:18:47-0800 [MyProtocol,55,127.0.0.1] launching subprocess
2009-12-22 11:18:47-0800 [MyProtocol,56,127.0.0.1] launching subprocess
2009-12-22 11:18:47-0800 [MyProtocol,55,127.0.0.1] launched subprocess, trying to communicate...
2009-12-22 11:18:47-0800 [MyProtocol,57,127.0.0.1]  launching subprocess
2009-12-22 11:18:47-0800 [MyProtocol,58,127.0.0.1] launching subprocess
2009-12-22 11:18:47-0800 [MyProtocol,56,127.0.0.1] launched subprocess, trying to communicate...
2009-12-22 11:18:47-0800 [MyProtocol,55,127.0.0.1] returning
2009-12-22 11:18:47-0800 [MyProtocol,57,127.0.0.1]  launching subprocess 
2009-12-22 11:18:47-0800 [MyProtocol,56,127.0.0.1]  launching subprocess returning
2009-12-22 11:18:47-0800 [MyProtocol,59,127.0.0.1]  launching subprocess
2009-12-22 11:18:47-0800 [MyProtocol,58,127.0.0.1]  launched subprocess, trying to communicate...
2009-12-22 11:18:47-0800 [MyProtocol,58,127.0.0.1] returning
2009-12-22 11:18:47-0800 [MyProtocol,59,127.0.0.1] launched subprocess, trying to communicate...
2009-12-22 11:18:47-0800 [MyProtocol,59,127.0.0.1] returning

MyProtocol, 57. , , " , ". , , , .

Answer 1

, . POSIX ( ) SIGCHLD , . SIGCHLD, . . Twisted (. http://twistedmatrix.com/documents/current/core/howto/process.html) Twisted, installSignalHandlers=False reactor.run ( , subprocess , Twisted).