Convert ast.Num to decimal.Decimal for precision in python

Question

I am currently writing a parser for parsing a simple arithmetic formula: it needs (and limits) support + - * / in terms of number and variables. For instance:

100.50*num*discount 

It is mainly used to calculate product prices.

It is written in python, and I would just like to use my own python parser for simplicity. The idea is to first analyze the input in ast and then switch to ast to restrict the type of ast node to a small subset, for example: ast.BinOp , ast.Add , ast.Num , ast.Name , etc. .

Currently it works well, except that the number of floating-point points in ast is inaccurate. So I want to convert ast ast.Num node to some ast.Call(func=ast.Name(id='Decimal'), ...) . But the problem is this: ast.Num contains only the field n , which is already the processed number of floating point points. And getting the source numeric literal in the source code is not easy: How to get the source corresponding to Python AST node?

Are there any suggestions?

Answer 1

I would suggest a two-step approach: in the first step, use the Python tokenize module to convert all floating-point numeric literals to source in the form string 'Decimal(my_numeric_literal)' . Then you can work on AST in the way that you propose.

There's even a recipe for the first step in the tokenize module. To avoid answering only the link, here is the code from this recipe (along with the necessary import, which the recipe itself is missing):

 from cStringIO import StringIO from tokenize import generate_tokens, untokenize, NAME, NUMBER, OP, STRING def is_float_literal(s): """Identify floating-point literals amongst all numeric literals.""" if s.endswith('j'): return False # Exclude imaginary literals. elif '.' in s: return True # It got a '.' in it and it not imaginary. elif s.startswith(('0x', '0X')): return False # Must be a hexadecimal integer. else: return 'e' in s # After excluding hex, 'e' must indicate an exponent. def decistmt(s): """Substitute Decimals for floats in a string of statements. >>> from decimal import Decimal >>> s = 'print +21.3e-5*-.1234/81.7' >>> decistmt(s) "print +Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7')" >>> exec(s) -3.21716034272e-007 >>> exec(decistmt(s)) -3.217160342717258261933904529E-7 """ result = [] g = generate_tokens(StringIO(s).readline) # tokenize the string for toknum, tokval, _, _, _ in g: if toknum == NUMBER and is_float_literal(tokval): result.extend([ (NAME, 'Decimal'), (OP, '('), (STRING, repr(tokval)), (OP, ')') ]) else: result.append((toknum, tokval)) return untokenize(result) 

The original recipe identifies floating point literals by checking for the existence of the value '.' in meaning. This is not entirely bulletproof since it excludes literals like '1e10' and includes imaginary literals like 1.0j (which you can exclude). I replaced this check with my own version in is_float_literal above.

Having tried this in your example line, I get the following:

 >>> expr = '100.50*num*discount' >>> decistmt(expr) "Decimal ('100.50')*num *discount " 

... which you can now analyze in the AST tree as before:

 >>> tree = ast.parse(decistmt(expr), mode='eval') >>> # walk the tree to validate, make changes, etc. ... >>> ast.dump(tree) "Expression(body=BinOp(left=BinOp(left=Call(func=Name(id='Decimal', ... 

and finally rate:

 >>> from decimal import Decimal >>> locals = {'Decimal': Decimal, 'num': 3, 'discount': Decimal('0.1')} >>> eval(compile(tree, 'dummy.py', 'eval'), locals) Decimal('30.150')