Python: can dumpdata not load data back. UnicodeDecodeError

Question

I have been using Python 2.7, Django 1.5, and PostgreSQL 9.2 for two weeks. Never seen this before. Everything has recently been installed on my computer with Windows 7, so it should have the default settings. Django nicely creates tables in my db. Everything seems to be working fine. I can dump data from my database by doing:

manage.py dumpdata > test.json 

or

 manage.py dumpdata --indent4 > test.json 

I saw that the JSON file looks as it should.

Then I truncate some tables and try to load them from a JSON file with:

 python manage.py loaddata database = T2 test.json // or without db name 

I got the following error:

"UnicodeDecodeError: codec 'utf8' cannot decode byte 0xff at position 0: invalid start byte"

If I open the test.json file in notepad, save it as utf8 and try again, then I will get:

"JSON object cannot be decoded"

The file still looks normal and not empty.

By the way, when I open the JSON file using notepad, it offers me to save it as Unicode. My database is UTF8 encoded. Please advise. Thanks.

Answer 1

0xff at position 0 looks like the beginning of a low- 0xff UTF-16 byte order marker for me. Notepad "Unicode" save mode is a little continental UTF-16, so it makes sense if you saved json from Notepad after creating it. Notepad will retain the byte order marker even in utf-8, which may cause loaddata to fail to parse it.

If you do not have an unmanaged json file, you need to delete the specification - I will personally use emacs, but another answer suggested this standalone Windows.exe:

http://www.bryntyounce.com/filebomdetector.htm