UnicodeError error when calling Django i18n makemessages command

Question

I work with Django internationalization features to create translation strings for webapp.

There is a problem when I try to call makemessages , and the existing .po language file contains a special character (e.g. $ , £ , etc.).

If one of them exists, makemessages tries to load the existing .po file and decode it. When he does this, I get an error message:

 Traceback (most recent call last): File "manage.py", line 18, in <module> execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv self.execute(*args, **cmd_options) File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 445, in execute output = self.handle(*args, **options) File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/makemessages.py", line 325, in handle self.write_po_file(potfile, locale) File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/makemessages.py", line 458, in write_po_file msgs, errors, status = gettext_popen_wrapper(args) File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/makemessages.py", line 51, in gettext_popen_wrapper stdout = stdout.decode(stdout_encoding) File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeEncodeError: 'ascii' codec can't encode character u'\xa2' in position 2105: ordinal not in range(128) 

I tried to get back on the track, but I do not understand what is happening.

It seems that Django is trying to decode the existing .po file as UTF8 , but then when re-encoding it uses the ASCII codec.

Any ideas as to what is wrong will be widely appreciated.

---

Edit:

• OS: Ubuntu 15.10 and OS X 10.11.6
• Python: 2.7.10 and 2.7.11
• Django: 1.8.14
• Six: 1.10.0

I tried reinstalling Django / Six as suggested, but the error still exists.

Ubuntu localedef --list-archive :

 en_AG en_AG.utf8 en_AU.utf8 en_BW.utf8 en_CA.utf8 en_DK.utf8 en_GB.utf8 en_HK.utf8 en_IE.utf8 en_IN en_IN.utf8 en_NG en_NG.utf8 en_NZ.utf8 en_PH.utf8 en_SG.utf8 en_US.utf8 en_ZA.utf8 en_ZM en_ZM.utf8 en_ZW.utf8 

Content type of the problem translation file:

  "Content-Type: text/plain; charset=UTF-8\n" 

Answer 1

Please note that this is another exception to this similar question mentioned in the comments.

It seems to me that the only way this can happen is to modify your django installation or an error in python version 2.7.

Your stack:

 > msgs, errors, status = gettext_popen_wrapper(args) > stdout = stdout.decode(stdout_encoding) 

gettext_popen_wrapper (on django 1.8 , what I think you are using, can you confirm?) and popen_wrapper which creates stdout (after deleting comments / docstrings and reindenting for clarity see popen_wrapper and gettext_popen_wrapper in github for unconfirmed code):

 def popen_wrapper(args, os_err_exc_type=CommandError, universal_newlines=True): try: p = Popen(args, shell=False, stdout=PIPE, stderr=PIPE, close_fds=os.name != 'nt', universal_newlines=universal_newlines) except OSError as e: strerror = force_text(e.strerror, DEFAULT_LOCALE_ENCODING, strings_only=True) six.reraise(os_err_exc_type, os_err_exc_type('Error executing %s: %s' % (args[0], strerror)), sys.exc_info()[2]) # NB: subprocess.Popen.communicate() should return two bytes # (ie str in python 2) objects output, errors = p.communicate() return ( output, force_text(errors, DEFAULT_LOCALE_ENCODING, strings_only=True), p.returncode ) def gettext_popen_wrapper(args, os_err_exc_type=CommandError, stdout_encoding="utf-8"): manual_io_wrapper = six.PY3 and stdout_encoding != DEFAULT_LOCALE_ENCODING stdout, stderr, status_code = popen_wrapper( args, os_err_exc_type=os_err_exc_type, universal_newlines=not manual_io_wrapper) if manual_io_wrapper: stdout = io.TextIOWrapper(io.BytesIO(stdout), encoding=stdout_encoding).read() if six.PY2: # EXCEPTION HIT ON THE FOLLOWING LINE stdout = stdout.decode(stdout_encoding) return stdout, stderr, status_code 

So, stdout should be a simple str object (i.e. a bunch of bytes that need decoding), by the time we call stdout.decode() . However, if this is so, then why is this an exception encoded in ru ? We would only need to encode if the object was already a unicode object, i.e. If it is of unicode type. And of course, if we add a line

 stdout = stdout.decode('utf-8') 

before

 stdout = stdout.decode(stdout_encoding) 

Then, now the decode method tries to encode unicode stdout using the standard ascii encoding , which throws the exception you saw. I also got the same error by setting manual_io_wrapper to True , which also caused the line stdout = io.TextWrapper(...) (which also creates Unicode), but it should not be True because you are on python 2 not 3.

Therefore, I think:

• You have a bad installation of django or six , or it has been edited. Try reinstalling them.
• You got a mistake in subprocess.Popen.communicate() and for some reason you are returning unicode not a str (I believe in python 3, which is possible if universal_newlines . You can get mileage by reinstalling python or updating it to a later version .

My main point is that I do not think this is a problem of the environment. It would be interesting to know about any follow-up actions:

• what platform are you on.
• which python 2.7 are you using
• which django you use.

Answer 2

On the next line, somehow, stdout not a str byte, but its unicode , and you get an exception while implicitly encoding that unicode .

 stdout = stdout.decode('utf-8') 

This is because decode() must be executed by str byte, and when we try to call decode on unicode , in python 2.7 there will be an implicit encode call with unicode before decode , and this encode call will use charset by default, which is ascii in python.

 unicode.encode() --> byte # results in str byte.decode() --> unicode # results in unicode unicode.decode() --> unicode.encode().decode() # implicit encode call 

So, start your research with the fact that the reason for stdout is unicode .

Thanks.