Manage.py datadump large database in Django 1.1 (creating a database)

Question

To be able to do some decent unittests, I want to create a testdatabase. After reading the Django docs, I came to the conclusion that the best way to do this is to create a test database from a real database using fixtures.

To create such a device, you need to run:

./manage.py dumpdata appname --indent 2

After dumping, I want to edit the file and select the data that I really want to use. However, resetting it almost crashed my PC (the scenarios are starting to figure out, I think). In addition, it seems to first get all the data in memory, and then print everything at once.

I am using a MySQL database

Is there a way to get data from the database without crashing my PC, in a format that can be used as a fixture?

(the alternative I'm considering is overriding the standard datamanager for objects with a lot of rows to return only the rows you need, but this seems like a pretty nasty hack that I would prefer not to use)

Answer 1

You should look at the ticket and apply the attached patch. @Ramiro recently ran tests to compare the trunk with its latest patch, and you can see that there is a huge difference in memory consumption due to processing time: