Json.dumps (pickle.dumps (u'å ')) raises a UnicodeDecodeError

Question

This is mistake?

>>> import json
>>> import cPickle
>>> json.dumps(cPickle.dumps(u'å'))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/encoder.py", line 361, in encode
    return encode_basestring_ascii(o)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-3: invalid data

Answer 1

The json module expects a string to encode text. The pickled data is not text, it is 8-bit binaries.

One easy way to work around the solution, if you really need to send pickled data via JSON, is to use base64:

j = json.dumps(base64.b64encode(cPickle.dumps(u'å')))
cPickle.loads(base64.b64decode(json.loads(j)))

, Python. 0 ASCII, å -ASCII- \xe5 , "\u00E5". - . http://bugs.python.org/issue2980

Answer 2

. python ( ): Protocol version 0 is the original ASCII protocol and is backwards compatible with earlier versions of Python. [...] If a protocol is not specified, protocol 0 is used.

>>> cPickle.dumps(u'å').decode('ascii')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 1: ordinal not in range(128)

aint no ASCII

, , :

 
>>> cPickle.dumps(u'å') == pickle.dumps(u'å')
False

Answer 3

I am using Python2.6 and your code works without errors.

In [1]: import json

In [2]: import cPickle

In [3]: json.dumps(cPickle.dumps(u'å'))
Out[3]: '"V\\u00e5\\np1\\n."'

By the way, what is your default system encoding, in my case it is

In [6]: sys.getdefaultencoding()
Out[6]: 'ascii'