Unable to enter unicode in python IDE (Mac OS X)

Question

I am trying to compile some unicode raw_input in the python default IDE, and as far as I know, this should be as simple as:

>>> c = raw_input() 日本語 >>> print c 日本語 

However, when I try to enter Unicode characters, the computer beeps for some protests, and I get an empty string. (To do this, I click on the IME switch near the time and select the appropriate input method [which in this case is the Japanese input). Outside of the python IDE, the input works fine, I can enter characters, and the system recognizes them as incoming. In the IDE, I type some hiragana and the window for selecting a descending kanji will appear, as usual, but when I select the appropriate view and press enter, these sound signals come and I end up with nothing. I suppose there was a place somewhere that I missed.

:

 Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin 

and

 Python 2.5.4 (r254:67916, Jun 24 2010, 21:47:25) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin 

none of them work. There also:

 >>> import sys >>> sys.getdefaultencoding() 'ascii' >>> sys.stdin.encoding 'UTF-8' >>> sys.stdout.encoding 'UTF-8' >>> sys.getfilesystemencoding() 'utf-8' 

but from what I read, the default coding is a mysterious beast. In any case, its change does not fix anything. I.e

 >>> import sys >>> sys.setdefaultencoding('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute 'setdefaultencoding' >>> reload(sys) <module 'sys' (built-in)> >>> sys.setdefaultencoding('utf-8') >>> # !!! ... c = raw_input() no dice! 

does not work. Sounds even more. I cannot cut and paste Japanese text from other applications.

Answer 1

I had the same problem. In my case, this turned out to be a libedit problem. I fixed this by installing readline - which I had to do from the source (from here: http://pypi.python.org/pypi/readline ), since pip or easy_install is being used , for some reason I did not actually replace readline.

If you have ipython installed, it will inform you of the startup if you are using libedit . And, if you have the same experience as me, you will see the same problems both in the python interpreter in the terminal and in ipython. Once I authenticated readline and ipython no longer informed me that it was using libedit, Unicode input issues disappeared in both python and ipython.

(Note: I also have bpython installed - and since it doesn’t seem to use readline or libedit, and its own Unicode line editing routines in bpython always worked.)

Answer 2

The default value should not affect this. I had a similar problem and for me the solution was to check the Escape non-ASCII option in terminal> Settings> Settings> Advanced. Also make sure the character encoding is set to Unicode (UTF-8) on the same settings page.

Answer 3

Edit: I tried Python from the command line (Terminal) and it does not work, and I get the sound signals you are talking about. This is apparently not a terminal limitation, since I can insert characters at the $ prompt in bash just fine. It works in Idle, as shown below.

Edit # 2: Interestingly, this one-liner works:

  $ python -c "exec(\"c=raw_input()\nprint c\")"日本語 <-- pasted日本語 

I would put this in a comment, but it would not format correctly. Output from 2.6.5 on MacOSX:

 Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 2.6.5 >>> c=raw_input()日本語>>> print c日本語>>> c u'\u65e5\u672c\u8a9e' >>> 

Answer 4

Try the following:

 import codecs, sys sys.stdin = codecs.getreader('UTF-8')(sys.stdin) sys.stdout = codecs.getwriter('UTF-8')(sys.stdout) sys.stderr = codecs.getwriter('UTF-8')(sys.stderr) print u'\u65e5\u672c\u8a9e' 

This works for me for non-ASCII characters when using Putty with the final encoding set to UTF-8. I see boxes because I don't have fonts for the installed CJK characters, but I think this should do it for you.

The reason for this is that, by default, the Python interpreter uses the ascii codec for stdin, stdout, and stderr. And since ASCII only defines byte values ​​from 0 to 127, only these byte values ​​can be printed.