The number of elements (models) in the database compared to the memory consumption of django RAM?

Question

I think this is a mistake in my own django code, but I just wanted to make sure.

Is the number of rows or models that I have in the database affect the memory consumption of django RAM? Suppose we had a mysql table called Model. My django code has only lazy request expectations in this form:

Models.objects.filter().blah().blah()[:SOME_NUMBER] 

SOME_NUMBER remains at a constant, so the data that is stored in memory is also constant and not related to the total number of "Models" (or at least I hope). However, on the My Processes tab, when the number of my models increases, RAM also increases. To the point where it gets too high!

Does this mean that this or something else leads to this? Besides queryset-db calls, what else can lead to high memory consumption and leaks? If nothing else, there should be something in my code where I request a bunch of models, and I don't know. My DJANGO_DEBUG settings are disabled.

Thanks.

Answer 1

There are a few things you should learn. Django does not guarantee to receive the most efficient request (and many times it does it pretty badly). The database cannot be configured correctly for the query you are sending, so it takes a long time to return the results.

---

What query is really sent to the database?
Use print str(MyModel.objects.filter(...).query) to view the constructed query. You can also enable DEBUG (which you should be in development, it's there for some reason) and run

 from django.db import connection print connection.queries 

How is this query evaluated in the database?
Copy SQL from above and run it directly in the database using explain <the query> . He will infer what is happening behind the scenes and how long it takes. This may show, for example, that you can add an index to a foreign key to increase the efficiency of the connection. The database may be fast enough with few rows, but a complex query can grow more than linearly with the number of rows.

---

You can also use the Django Debug toolbar to automatically get this information on the pages you are viewing.

Answer 2

If the problem is related to memory consumption, it is very difficult to predict from a small code example like this. To track down the problem, you probably need to use a memory profiler in your application.

This stack overflow question on Which Python memory optimizer is recommended gives a good, concise example using Heapy , which shows how you can print a memory dump at some point in your code and understand what type of objects the memory takes. This should give you a good starting point for tracking memory leaks.

Answer 3

Why don't you temporarily delete some rows in the above table or delete all the other tables to test your hypothesis?