All geek questions in one place

Python processes increase memory usage daily

Scenario:

I have a python process that runs continuously (until we stop manually - endlessly) and collect data by reading certain system files every 1 minute. When it is running, it takes about 25 MB. But day after day, the amount of occupied memory increases, and after 15 days it is more than 500 MB.

  • I am new to python, so I cannot find the problem by going through the code manually / using a debugger,
  • I do not know a single tool that can easily identify a problem, please suggest some tool or method that can help me in identifying a problem.
  • I suppose this could be a memory leak problem or a variable that adds data to it again and again, and does not free it at all ... Please give me some links that can help me in understanding these problems in python.

Hope the details are clear enough, let me know for more information, I can clarify.

Update

I tried to follow this topic Show stack trace from a running Python application , but I am encountering the following errors that cannot be completely resolved.

  • There is no "PyEval_EvalFrameEx" character in the current context.
  • There is no "PyStringObject" character in the current context.

I was looking for this method to connect to an already running process and get stack / memory information. But no luck, kindly help me.

+4
source share
3 answers

Use Dowser to track memory leaks. It uses CherryPy as a web server, and even if you are not developing a web application, it can provide you with an idea of ​​memory allocation from your browser.

See also this post that may interest you. With Dowser, you can see what unexpected objects hang in your process memory.

+3
source

This is a well-known issue, manifested in long python processes. Improvements were made, but did not completely solve the problem. For example, Python uses its own memory management algorithm, which can lead to fragmentation of its own heap. The best strategy is to start several processes and gracefully restart when each of them falls into a high memory watermark. Apache also uses this strategy.

+1
source

Overall: Python has problems with memory management - especially with freeing up memory. Rebooting applications regularly is a common practice. Also, learn about using the gc garbage collector and browse streams like

How to profile memory usage in Python?

-eleven
source

Source: https://habr.com/ru/post/1340662/

More articles:


All Articles