Automatically kill a process that consumes too much memory or stops on Linux

Question

I need a "system" that controls the process and kills the specified process if:

• process exceeds some memory requirements
• the process does not respond to a message from the "system" for some period of time

I suppose this “system” might be something simple, like a monitoring process? Sample code on how this could be done would be helpful. Of course, I would not mind a completely different solution to this problem.

Answer 1

For the first requirement, you can look either with ulimit , or with tuning the OOM-killer kernel settings on your system.

Demon monitoring also exists for this kind of thing. God is a recent example.

Answer 2

I wrote a script that runs as a cron job and can be configured to kill problem processes:

 #!/usr/local/bin/perl use strict; use warnings; use Proc::ProcessTable; my $table = Proc::ProcessTable->new; for my $process (@{$table->table}) { # skip root processes next if $process->uid == 0 or $process->gid == 0; # skip anything other than Passenger application processes #next unless $process->fname eq 'ruby' and $process->cmndline =~ /\bRails\b/; # skip any using less than 1 GiB next if $process->rss < 1_073_741_824; # document the slaughter (my $cmd = $process->cmndline) =~ s/\s+\z//; print "Killing process: pid=", $process->pid, " uid=", $process->uid, " rss=", $process->rss, " fname=", $process->fname, " cmndline=", $cmd, "\n"; # try first to terminate process politely kill 15, $process->pid; # wait a little, then kill ruthlessly if it still around sleep 5; kill 9, $process->pid; } 

http://blog.endpoint.com/2012/08/automatically-kill-process-using-too.html

Answer 3

To limit memory to processes, check / etc / security / limits.conf

Answer 4

Try Process Resource Monitor for a classic, easy-to-use process monitor. The code is available under the GPL.

There are also several other monitoring scenarios that may seem interesting.

Answer 5

If you want to create a fairly complete monitoring system, look at monit . Sometimes it can be very (VERY VERY VERY), but it will control a lot, restart the services, warn you, etc.

However, don't be surprised if you receive dozens of emails a day until you get used to setting it up and say what you don't need.

Answer 6

Are the monitored processes the ones you write, or just any process?

If these are arbitrary processes, it can be difficult to control the reaction. If the process is already configured to process and respond to events that you can send, I doubt that you can control them. If these are the processes you write, you need to add some kind of message processing with which you can use validation.