Unicode issue with makemessages - all Django 1.6.2 Python 3.3

Question

Project update with Python 2.7 → 3.3.1 and Django 1.4 → 1.6.2 .

After updating the code, our application starts up again (in py3).
Translations are derived from .mo files.

The only problem is that our old .po files cannot be used with

django-admin.py makemessages -a

It displays beautiful

UnicodeDecodeError: 'ascii' codec can't decode byte...

We can run makemessages for the first time and get the skeleton files. As soon as we add any non-ASCII translations (ǹ, è, etc.) to msgstr values, makemessages will not complete.
(If we run makemessages with higher word patterns with any non-ASCII characters, skipped.)

I found error reports for a similar problem, but they returned in version 1.3.x, but there was nothing for the above versions.

---

Update, additional information:

Here where the exception occurs:
.. / python 3.3 / subprocess.py line 847

 def _translate_newlines(self, data, encoding): data = data.decode(encoding) return data.replace("\r\n", "\n").replace("\r", "\n") 

The encoding value is ANSI_X3.4-1968 . I saved the template files as UTF-8 along with the .po file.

Here is the .po header (only skeleton created automatically from makemessages):

 # SOME DESCRIPTIVE TITLE. # Copyright (C) YEAR THE PACKAGE COPYRIGHT HOLDER # This file is distributed under the same license as the PACKAGE package. # FIRST AUTHOR < EMAIL@ADDRESS >, YEAR. # #, fuzzy msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" "POT-Creation-Date: 2014-02-28 22:42+0000\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME < EMAIL@ADDRESS >\n" "Language-Team: LANGUAGE < [e-mail] >\n" "Language: \n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" "Plural-Forms: nplurals=2; plural=(n != 1);\n" 

This worked before (same files) under Python 2.7 and Django 1.5

---

Update # 2

• List item
• New bare project created (django-admin.py startproject blah)
• Included i18n etc.
• Created a single translation (only in settings.py)
• Ran `makemessages -l de
• Py2.7 (#python manage.py makemessages -a) works as expected
• Py3.3 (# python3 manage.py makemessages -a) not working

Perhaps a bug will be fixed, will be updated.

Answer 1

This threw me away as everything was fine with Py2 but not Py3, so I assumed there would be a problem.

The problem was partly because I use Docker and run makemessages from a container that did not have the locale installed for anything, specifically for bash.

Things I tried:

• Saving files using UTF-8 (with and without specification)
• Ensuring UTF-8 in the header of .po files
• Create a new project with a blank slide
• Re-creating all .po files with Py3 (since they were originally created with Py2)

The top exception was sent to subprocess.py in this line 847:

 def _translate_newlines(self, data, encoding): data = data.decode(encoding) return data.replace("\r\n", "\n").replace("\r", "\n") 

The transmitted encoding was ANSI_X3.4-1968 , which was strange since I saved the files as UTF-8, etc. (it was installed as ANSI ... because of my bash session, no local set was installed).

Answer
In my Docker container, I did not have the locale settings installed in the terminal, so they were:

 # locale LANG= LANGUAGE= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= 

These were my available locales (my specific locale, en_US.UTF-8 , but so far it is UTF-8, I'm fine):

 # locale -a C C.UTF-8 POSIX 

Posted this in ~/.bashrc :

 export LC_ALL=C.UTF-8 export LANG=C.UTF-8 export LANGUAGE=C.UTF-8 

And now I get UTF-8 as the content type inside subprocess.py , and everything works with Py3 / Django1.6 =)

In short, I was caught that Django / subprocess.py uses the locale of the environment and not the encoding of the file / or Content-Type header.