How to debug error in `ast.literal_eval`?

Question

I wrote my data to a file using pprint.PrettyPrinter, and I'm trying to read it using ast.literal_eval. This has been working for me for quite some time, and I am quite satisfied with the created text view.

However, today I received this error while deserializing:

  File "/...mypath.../store.py", line 82, in <lambda>
    reader=(lambda fd: ast.literal_eval(fd.read())),
  File "/usr/lib64/python2.7/ast.py", line 80, in literal_eval
    return _convert(node_or_string)
  File "/usr/lib64/python2.7/ast.py", line 60, in _convert
    return list(map(_convert, node.elts))
  File "/usr/lib64/python2.7/ast.py", line 63, in _convert
    in zip(node.keys, node.values))
  File "/usr/lib64/python2.7/ast.py", line 62, in <genexpr>
    return dict((_convert(k), _convert(v)) for k, v
  File "/usr/lib64/python2.7/ast.py", line 63, in _convert
    in zip(node.keys, node.values))
  File "/usr/lib64/python2.7/ast.py", line 62, in <genexpr>
    return dict((_convert(k), _convert(v)) for k, v
  File "/usr/lib64/python2.7/ast.py", line 79, in _convert
    raise ValueError('malformed string')
ValueError: malformed string

How to fix this particular file?

This file is 17k lines / 700kb. I uploaded it to Emacs - balanced parens. There are no non-ASCII characters in the file. I can “split and win” (split the file in half and try every second), but it's pretty tedious. Anything better?

I changed ast.literal_eval:_convertto print the offending node - it turned out to be <_ast.UnaryOp object at 0x110696510>. Not very helpful.

How can I guarantee that this will not happen in the future?

, JSON .; -)

JSON,

• JSON
• JSON

Answer 1

:

--- /...../2.7/lib/python2.7/ast.py.old 2018-03-25 12:17:11.000000000 -0400
+++ /...../2.7/lib/python2.7/ast.py 2018-03-25 12:17:18.000000000 -0400
@@ -76,7 +76,7 @@ def literal_eval(node_or_string):
                 return left + right
             else:
                 return left - right
-        raise ValueError('malformed string')
+        raise ValueError('malformed string', node.lineno, node.col_offset)
     return _convert(node_or_string)

ast:

>>> reload(ast)

Get

ValueError: ('malformed string', 21161, 10)

21161, 10, .

try/except, inspect/traceback node :

try:
    ast.literal_eval(...)
except ValueError as ex:
    _exc_type, exc_value, exc_traceback = sys.exc_info()
    print("ERROR: %r" % (exc_value))
    # traceback.print_tb(exc_traceback)
    last_tb = exc_traceback
    while last_tb.tb_next:
        last_tb = last_tb.tb_next
    print("Error location: line=%d, col=%d" % (
        last_tb.tb_frame.f_locals["node"].lineno,
        last_tb.tb_frame.f_locals["node"].col_offset))

ERROR: ValueError('malformed string')
Error location: line=21933, col=15