OS starts destruction processes when python multithreaded process starts

Question

This is the strangest!

I have a multi-threaded client application written in Python. I use threading to simultaneously load and process pages. I would use the cURL multiroom, except that the bottleneck is certainly the processor (and not bandwidth) in this application, so it’s more efficient to use the thread pool.

I have a 64-bit i7-download 16-gigabyte RAM. Muscular. I run 80 threads listening to Pandora and trolling Stackoverflow and BAM! The parent process sometimes ends with a message

Killed

In other cases, one page (which is Chrome’s own process) will die. In other cases, the entire browser crashes.

If you want to see some code, here is the gist:

Here is the parent process:

 def start( ): while True: for url in to_download: queue.put( ( url, uri_id ) ) to_download = [ ] if queue.qsize( ) < BATCH_SIZE: to_download = get_more_urls( BATCH_SIZE ) if threading.activeCount( ) < NUM_THREADS: for thread in threads: if not thread.isAlive( ): print "Respawning..." thread.join( ) threads.remove( thread ) t = ClientThread( queue ) t.start( ) threads.append( t ) time.sleep( 0.5 ) 

And here is the essence of ClientThread:

 class ClientThread( threading.Thread ): def __init__( self, queue ): threading.Thread.__init__( self ) self.queue = queue def run( self ): while True: try: self.url, self.url_id = self.queue.get( ) except: raise SystemExit html = StringIO.StringIO( ) curl = pycurl.Curl( ) curl.setopt( pycurl.URL, self.url ) curl.setopt( pycurl.NOSIGNAL, True ) curl.setopt( pycurl.WRITEFUNCTION, html.write ) curl.close( ) try: curl.perform( ) except pycurl.error, error: errno, errstr = error print errstr curl.close( ) 

EDIT: Oh, right ... forgot to ask a question ... it should be obvious: why are my processes killed? Does this happen at the OS level? Core level? Is this due to the limitation on the number of open TCP connections that I have? Is this a limit on the number of threads that I can start right away? The output of cat /proc/sys/kernel/threads-max is 257841 . So ... I do not think that ....

I think I did it ... OK ... I do not have swap space on my disk. Is there a way to create some swap space? I run Fedora 16. There WAS swap ... then I turned on all of my RAM and it disappeared magically. Tailing /var/log/messages I found this error:

 Mar 26 19:54:03 gazelle kernel: [700140.851877] [15961] 500 15961 12455 7292 1 0 0 postgres Mar 26 19:54:03 gazelle kernel: [700140.851880] Out of memory: Kill process 15258 (chrome) score 5 or sacrifice child Mar 26 19:54:03 gazelle kernel: [700140.851883] Killed process 15258 (chrome) total-vm:214744kB, anon-rss:70660kB, file-rss:18956kB Mar 26 19:54:05 gazelle dbus: [system] Activating service name='org.fedoraproject.Setroubleshootd' (using servicehelper) 

Answer 1

You have activated the Out Of Memory (OOM) kernel handler; he chooses which processes to kill in a complicated way, which tries to kill as few processes as possible in order to make the greatest impact. Chrome, apparently, makes the process of killing by the criteria used by the kernel the most attractive.

You can view a summary of the criteria in the proc(5) wizard in the /proc/[pid]/oom_score :

  /proc/[pid]/oom_score (since Linux 2.6.11) This file displays the current score that the kernel gives to this process for the purpose of selecting a process for the OOM-killer. A higher score means that the process is more likely to be selected by the OOM- killer. The basis for this score is the amount of memory used by the process, with increases (+) or decreases (-) for factors including: * whether the process creates a lot of children using fork(2) (+); * whether the process has been running a long time, or has used a lot of CPU time (-); * whether the process has a low nice value (ie, > 0) (+); * whether the process is privileged (-); and * whether the process is making direct hardware access (-). The oom_score also reflects the bit-shift adjustment specified by the oom_adj setting for the process. 

You can configure the oom_score file for your Python program if you want it to be killed.

Probably the best approach is to add more swaps to your system in order to try to push away the time when the OOM-killer is called. Of course, having more swaps does not necessarily mean that your system will never run out of memory - and you might not like how it is handled if there is a lot of swap traffic, but it can at least get you memory problems in the past.

If you have already allocated all the space for swap partitions, you can add swap files. Since they go through the file system, there is more overhead for swap files than swap partitions, but you can add them after the drive is partitioned, making it an easy short-term solution. You use the dd(1) command to highlight the file ( do not use seek to create a sparse file ), and then use mkswap(8) to format the file to use swap, then use swapon(8) to include this particular file. (I think you can even add swap files to fstab(5) so that they are automatically available the next time you reboot too, but I never tried and don't know the syntax.)

Answer 2

you do

 raise SystemExit 

which actually terminates the Python interpreter, not the thread you are working on.