Simple Python 3 templating module

The module adds syntactic sugar to Python 3 to allow concise specification of templates. I use it to generate a collection of static web pages from various data sources.

I wanted something relatively simple without the overhead of, say, cheetah or more recent templating systems (e.g. Mako and Jinja2). (I don't want to separate the templating logic from the templates, nor do I need direct integration with a web server. I just want to be able to maintain a collection of static web pages built from an occasionally changing source of data.)

The approach for this module is to extend Python 3 with just a few elements of syntactic sugar that help specify templates in python particularly concisely. But to do that, the module does several hackish things: it hooks into the import system, and it rewrites the abstract syntax trees of imported template files to implement the syntactic sugar.

I know the code is currently rough, error handling is rudimentary. I'm mainly looking for feedback on the overall approach. For example, this approach intentionally sacrifices modularity and the principle of "making things explicit", to obtain convenience and conciseness in specifying templates. Is there a way to obtain those benefits without those sacrifices? I'd also like to know of similar existing modules.

The code is hosted at Github. Here is the package structure:

.
├── __init__
├── compile
├── gather
└── load

The compile, gather, and load files are private, they cannot be imported by the user. The package itself is used solely through import template or import template.xxx statements, as described in the doc string in __init.py and the readme (the readme is more readable). Here is the code:

File __init__.py

```
#!/usr/bin/env python3

'''Syntactic sugar for convenient template generation in python3.

The "template" module tweaks the python3 import system so that (assuming
the template module is on the python import path) the statement

There's a lot of code here, so I'm just going to look at the documentation, the _split_by_braces function, and the visit_Str method. You'll see that there's more than enough for one answer. Maybe some of the other users here will comment on the other parts of your program.

1. Documentation

-
Most submissions to Code Review have no documentation at all (requiring us to reverse-engineer everything from code), so to see comprehensive documentation like this is excellent.

-
However, it could be improved. What you have here is reference documention: organized by features. But programmers also need user documentation, organized by use cases. A programmer starts by approaching a piece of software with a task: "how do I turn a list of objects into an HTML table?" and only later do they start asking questions about the exact behaviour of comments in nested templates. And most people learn more quickly from examples than from specifications.

So I would recommend starting the documentation with examples and use cases. Take a look at the Jinja2 and Django Template Language introductions: they both start with examples.

-
The examples are not very motivating. It is hard to see why I would want to output a string like:

1 Render a=X, f()=F X B.

Good examples make it easier to understand how a software system works, and just as importantly, why someone would want to use it. It's hard to come up with examples that are clear and succinct, but worth the effort. You can see that the Django docs do a slightly better job of this: their first example is a news aggregator (specific and practical), whereas Jinja2's first example is a generic "My Webpage" with navigation.

-
The examples are not automatically checkable: how would you know if they no longer worked? Changing them into doctests would solve this.

1. _split_by_braces

-
There's an example in the docstring that could be turned into a doctest so that it can be automatically checked.

-
If you do this, then you'll see that the example has a mistake:

Failed example:
    list(_split_by_braces('aa{{b{{c}} }}d{{e}}'))
Expected:
    ['aa', ' b{{c}} ', 'd', 'e', '']
Got:
    ['aa', 'b{{c}} ', 'd', 'e', '']

-
There is no need to misspell "string" as "strng" to avoid shadowing the string module, since you don't use that module in this code.

-
The docstring says:


For each piece, return (depth, piece).

but this is not right (presumably this was true in an earlier version of the code, but then you realised that if depth always alternates then it's superfluous to return it). It's worth getting into the habit of checking the docstring every time you make a change to a function (or better still, change the documentation first).

-
Since you're reusing the same regular expression each time the function is called, consider compiling it just once.

-
It's a bad idea to raise a plain Exception for specific errors (like unbalanced format strings). How can a caller catch just this specific error and no others? Exception is the root of the built-in exception hierarchy, so if a caller tries to catch it, they catch all built-in, non-system-exiting exceptions too.

It is better to define a specific exception class:

class UnbalancedError(Exception):
    pass

-
When reporting an error involving an invalid string, it's best to give the repr of the string (as the string itself might include whitespace or newlines, which would be confusing). Python's built-in error messages look like this:

>>> int('abc')
Traceback (most recent call last):
  File "", line 1, in 
ValueError: invalid literal for int() with base 10: 'abc'

so you could use something like this:

raise UnbalancedError("too many close braces: {!r}".format(string))

-
The final error condition is:

if depth > 0:

(because you are confident that depth

Revised code:

``
class UnbalancedError(Exception):
pass

_BRACE_PAIR_RE = re.compile("{{|}}")

def _split_by_braces(string):
'''Generator that splits string into pieces separated by top level
pairs of braces, and yields the pieces.

>>> list(_split_by_braces('aa{{ b{{c}} }}d{{e}}'))
['aa', ' b{{c}} ', 'd', 'e', '']

The pieces alternate between unbraced and braced, with the first
and last pieces unbraced. This may require some of the pieces to
be empty strings to maintain the alternation:

>>> list(_split_by_braces('{{a}}{{b}}{{c}}'))
['', 'a', '', 'b', '', 'c', '']

Raises UnbalancedError if the pairs of braces are not properly
nested:

>>> list(_split_by_braces('{{{{}}'))
... # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
UnbalancedError: too many open braces: '{{{{}}'

'''
start = depth = 0
for match in _BRACE_PAIR_RE.finditer(string):
prev_depth = depth
depth += 1 if match.group(0) == '{{' else -1
if depth < 0:
raise UnbalancedError("too many close braces: {!

StackExchange Code Review Q#109726, answer score: 11