How do I know if a running script is dying?

Question

So, I'm a little new to programming and mostly self-taught, so sorry if this question is a bit on the side of newbies.

I have a python script that runs for a long time (for example, it loads pages every few seconds for several days in a row). Sort monitoring script for web application.

Every time something so often violates it, and it needs to be restarted. I got these events at the minimum minimum, but it still happens every few days, and when he is killed, this may be bad news if I don’t notice for several hours.

He is currently working on a screen session on VPS.

Can someone point me in the right direction, as far as I know, when the script dies / and automatically restarts?

Will it write something in Bash? Or something different? I have never done anything like this before and I don’t know where to start or even look for information.

Answer 1

I believe that the bash script wrapper that runs the python script inside the loop should do the trick.

while true; do # Execute python script here echo "Web app monitoring script disrupted ... Restarting script." done 

Hope this helps.

Answer 2

You can try supervisord , it is a daemon process control tool.

Answer 3

It depends on the type of failure you want to protect against. If this is just a script crash, the simplest thing would be to wrap your main function in try / except:

 import logging as log while True: try: main() except: log.exception("main() crashed") 

If something kills the Python process, the easiest way to start it is in a shell loop:

 while sleep 1; do python checker.py; done 

And if it collapses because the car goes down ... well ... Quis custodiet ipsos custodes?

However, to answer your question directly: the easiest way to check if it works from the shell is to grep output ps :

 ps | grep "python checker.py" 2>&1 > /dev/null running=$? 

Of course, this is not perfect, but overall it is good.

Answer 4

You must demonize your program.

As described in the Effective Python daemon , you can install and use python-daemon , which implements the standard specification of the PEP 3143 daemon, "Library of the standard daemon."

Create a mydaemon.py file with the following contents:

 #!/usr/bin/env python import daemon import time import logging def do_something(): name = 'mydaemon' logger = logging.getLogger(name) handler = logging.FileHandler('/tmp/%s.log' % (name)) formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s') handler.setFormatter(formatter) logger.addHandler(handler) logger.setLevel(logging.WARNING) while True: try: time.sleep(5) with open("/tmp/file-does-not-exist", "r") as f: f.write("The time is now " + time.ctime()) except Exception, ex: logger.error(ex) def run(): with daemon.DaemonContext(): do_something() if __name__ == "__main__": run() 

To run it, use:

 python mydaemon.py 

What do_something() will appear in DaemonContext , and then mydaemon.py script will exit. You can see the running daemon with: pgrep -fl mydaemon.py . This short example simply writes errors to the log file in /tmp/mydaemon.log . You will need to kill the demon manually or it will work endlessly.

To run your own program, simply replace the contents of the try block with a call to your code.