I seem to have a memory leak in a Google App Engine application, but I can't figure out why.
After narrowing the line of code responsible for this, I reduced the problem to a simple cron job that runs regularly, and all it does is load some objects using the query.
I turn on memory usage using logging.info(runtime.memory_usage()) , and I see that memory usage increases from one call to another until it exceeds the soft personal memory limit.
Below is the code I used:
class User(ndb.Model): _use_cache = False _use_memcache = False name = ndb.StringProperty(required=True) ...[OTHER_PROPERTIES]... class TestCron(webapp2.RequestHandler): def get(self): is_cron = self.request.headers.get('X-AppEngine-Cron') == 'true' if is_cron: logging.info("Memory before keys:") logging.info(runtime.memory_usage()) keys = models.User.query().fetch(1000, keys_only=True) logging.info("Memory before get_multi:") logging.info(runtime.memory_usage()) user_list = ndb.get_multi(keys) logging.info("Memory after:") logging.info(runtime.memory_usage()) logging.info(len(user_list)) app = webapp2.WSGIApplication([ ('/test_cron', TestCron) ], debug=True)
And in cron.yaml I have:
- description: Test cron url: /test_cron schedule: every 1 mins from 00:00 to 23:00
When performing this task every minute every 2 iterations, you need to restart a new instance. The first time starts with 36mb, and upon completion it says
This request triggered a new process for your application and thus caused the loading of your application code for the first time. Thus, this request may take longer and use more CPU than the typical request for your application.
But on the second run, it starts by using 107 MB of used memory (which means that it did not clear the memory from the previous iteration?), And it exceeds the limit on soft personal memory and ends the process, saying:
Exceeding the limit limit of private memory 128 MB from 134 MB after serving only 6 requests After processing this request, it was discovered that the process processing this request was using too much memory and was interrupted. This will likely cause the new process to be used for the next request to your application. If you often see this message, you may have a memory leak in your application.
The following is the full output of both logs:
The result simply alternates between the two logs. Please note that I disabled the cache in the model definition, so shouldn't the memory usage be reset every time the function is called?