How to speed up iteration of large datasets in Django

Question

I have a query set of approximately 1,500 records from a Django ORM query. I used the select_related () and only () methods to make sure the request is tight. I also used connection.queries to make sure there is only this request. That is, I made sure that additional requests are called for each iteration.

When I run a query cut and pasted from connection.queries, it starts in 0.02 seconds. However, it takes seven seconds to repeat these entries and do nothing with them (skip).

What can I do to speed this up? What causes this slowness?

Answer 1

QuerySet can become quite heavy when it is filled with model objects. In situations like this, I used the .values ​​method for a set of queries to specify the properties that I need, as a list of dictionaries, which can be much faster to repeat. http://docs.djangoproject.com/en/1.3/ref/models/querysets/#values-list

Answer 2

1,500 records are far from a large dataset, and seven seconds is really too much. There are probably some problems with your models, you can easily verify this by receiving (as Brandon says) a values ​​() request, and then explicitly create the 1500 object by iterating the dictionary. Just convert ValuesQuerySet to a list before construction to split the db connection.

Answer 3

How do you iterate over each element:

items = SomeModel.objects.all() 

Regularly for each cycle

 for item in items: print item 

Or using a QuerySet iterator

 for item in items.iterator(): print item 

According to the document, iterator() can improve performance. The same applies when looping through a very large list or Python dictionaries, iteritems() best used.

Answer 4

Does your Meta declaration indicate a “sort by" field that is stored in some other related table? If so, your attempt to repeat can trigger 1,500 requests, as Django runs away and grabs this field for each element, and then sorts them. Showing that your code will help us understand the problem!