Calculator using tokens of numbers and operators

Lately there has been some questions regarding calculators, where the response typically has suggested simplifying the logic and using either abstract syntax tree or token lists (like I did myself in a Java calculator question).

To prove the concept, and as a training exercise for my self I implemented this calculator (in Python 2.x), as a proof concept, and to visualize one way of implementing a calculator which supports operator precedence, exponentiation and floor division.

Here is the code:

```
# -- coding: utf-8 --
from __future__ import division, print_function
from textwrap import dedent
import math
import operator as op
from collections import OrderedDict
import re
import doctest

# Regex to split into numbers and operators
TOKEN_SPLITTER = re.compile("""
\s* # Ignore leading whitespace
( # Group the result, which is either ...
(?>> tokenizer("10-2^3*5/2--5")
[10, '-', 2, '^', 3, '*', 5, '/', 2, '-', -5]

>>> tokenizer("1*2.0/3**4+5-6")
[1, '*', 2.0, '/', 3, '**', 4, '+', 5, '-', 6]

>>> tokenizer(" 2 * -3 + 6 /3--5 ")
[2, '*', -3, '+', 6, '/', 3, '-', -5]
"""

def _numberify(token):
"""Turn token into int or float, if possible."""

try:
# Check if it is an int and return it
return int(token)

except ValueError:
# Not an int, is it possibly a float?
try:
return float(token)

except ValueError:
# Not a number, return operator as is
return token

## Split into numbers and operators, and make number strings into numbers
return map(_numberify, TOKEN_SPLITTER.findall(expression))

def calculate(expression):
"""Calculate the value of the list or expression.

The function can be called with a string expression, or a prebuilt
list of tokens from tokenizer(). It will apply the calculations
according to precedence of ope

Not exactly an "improvement" to your code, but this comment:

# '(' and ')' are not implemented yet!, and needs special care

I would change this to:

# TODO: '(' and ')' are not implemented yet!, and needs special care

It is an extremely small detail, but there are a lot of reasons why. My editor is set up to highlight the word TODO, similarly, it is now easy to search through your code and determine where work needs to be done. It doesn't matter too much for this small project, but for larger projects you'll frequently find the TODO throughout a project.

StackExchange Code Review Q#114687, answer score: 7