Unicode Python runs on version 2.6.1 on OSX, but not on 2.6.5 on Ubuntu

Question

Given the following code executed by the Python interpreter:

import sys
sys.getdefaultencoding()
my_string = '\xc3\xa9'
my_string = unicode(my_string, 'utf-8')
my_string
print my_string

With Python 2.6.1 running on a Mac, everything works fine:

$ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> my_string = '\xc3\xa9'
>>> my_string = unicode(my_string, 'utf-8')
>>> my_string
u'\xe9'
>>> print my_string
é
>>> 

When Python 2.6.5 runs on Ubuntu 10.04 LTS, it fails:

$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> my_string = '\xc3\xa9'
>>> my_string = unicode(my_string, 'utf-8')
>>> my_string
u'\xe9'
>>> print my_string
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0: ordinal not in range(128)
>>> 

Has something changed between Python 2.6.1 and 2.6.5 that requires different string handling in unicode? Or is it due to something incorrectly configured in my (default Ubuntu server 10.04 LTS) Linux environment?

Edit: both environments have LANG = en_US.UTF-8

Answer 1

I can reproduce the error with the command:

$ PYTHONIOENCODING=ascii python -c'print "\xc3\xa9".decode("utf-8")'

Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0:\
ordinal not in range(128)

sys.getdefaultencoding() 'ascii' .

:

$ PYTHONIOENCODING=utf-8 python -c'print "\xc3\xa9".decode("utf-8")'
é

$ python -c'import locale; print "\xc3\xa9".decode("utf-8").encode(
> locale.getpreferredencoding())'
é

sys.stdout.encoding:

$ python -c'import sys; o = sys.stdout; print o.isatty(), o.encoding'
True UTF-8

$ python -c'import sys; o = sys.stdout; print o.isatty(), o.encoding' | cat
False None

$ python -c'import sys; o = sys.stdout; print o.isatty(), o.encoding' >/tmp/out
$ cat /tmp/out
False None

sys.stdout.encoding None, locale.getpreferredencoding() PYTHONIOENCODING, . . http://wiki.python.org/moin/PrintFails

Python, sys.displayhook().

Answer 2

C. Python LANG=en_US.UTF-8 python .

Answer 3

u?

my_string = u '\ xc3\xa9'

. http://docs.python.org/howto/unicode.html#unicode-literals-in-python-source-code

In the Python Unicode source code, literals are written as strings with the prefix "u" or "U" character: u'abcdefghijk '. A specific code point can be written using the \ u escape sequence, followed by four hexadecimal digits giving the code point. The escape sequence \ U is similar, but expects 8 hexadecimal digits, not 4.