I need to use django reset_queries ()

Question

I am using django 1.3, and I am running the script out of web context (from the command line).
my code keeps reading 10,000 records from db every time.
I noticed that the use of memory in the process becomes more and more with time.
my code is:

def getData(startIndex,chunkSize): dataList =Mydata.objects.filter(update_date__isnull = True)[startIndex:startIndex+chunkSize] return list(dataList) if __name__ == '__main__': chunkSize = 10000 startIndex = 0 dataSize = Mydata.objects.filter(update_date__isnull = True).count() while startIndex < dataSize: dataList = getData(startIndex,chunkSize) startIndex += chunkSize do_stuff(dataList) 

my question is: do I need to use reset_queries() and or connection.close()
and is this the reason for the increase in memory usage?

Answer 1

I would start by using only or defer in your request. These two are used to get only the fields that you really need, instead of all the fields. Your query will be a little faster and consume less memory, because unnecessary fields will not be retrieved from the database.