How can I load a Django device too large to fit in memory?

Question

I want to load source data using fixtures as described here

https://docs.djangoproject.com/en/dev/howto/initial-data/

That would be easy enough with a small dataset. However, I want to load a large CSV that will not fit into memory. How can I serialize this to a large JSON format? Do I have to hack it by manually opening '[' and closing ']' or is there any way to do this?

Answer 1

When you see that you start with a CSV file, you can create a custom command. You can read the CSV file, create the objects and save them to the database inside this command. As long as you can process each CSV line in a loop, you will not run into memory problems.

Relative documentation can be found here:

http://docs.python.org/2/library/csv.html https://docs.djangoproject.com/en/dev/howto/custom-management-commands/