Controlling the order of unittest.TestCases

Apparently everyone gets burned by their Python unittests not running in the order they want.

I am not in the business of telling people not to do perfectly reasonable things they want to do, so I consider "write your tests differently" Not An Answer to the question "how do I control the order of my TestCase subclasses".

With that in mind, I also consider "Why do you want to control the order of unittests? Just write them differently" as a lame, non-answer to this CR post.

I do, however, consider the following (more than) an answer to the above question:

```
import unittest

def suiteFactory(
*testcases,
testSorter = None,
suiteMaker = unittest.makeSuite,
newTestSuite = unittest.TestSuite
):
"""
make a test suite from test cases, or generate test suites from test cases.

*testcases = TestCase subclasses to work on
testSorter = sort tests using this function over sorting by line number
suiteMaker = should quack like unittest.makeSuite.
newTestSuite = should quack like unittest.TestSuite.
"""

if testSorter is None:
ln = lambda f: getattr(tc, f).__code__.co_firstlineno
testSorter = lambda a, b: ln(a) - ln(b)

test_suite = newTestSuite()
for tc in testcases:
test_suite.addTest(suiteMaker(tc, sortUsing=testSorter))

return test_suite

def caseFactory(
scope = globals().copy(),
caseSorter = lambda f: __import__("inspect").findsource(f)[1],
caseSuperCls = unittest.TestCase,
caseMatches = __import__("re").compile("^Test")
):
"""
get TestCase-y subclasses from frame "scope", filtering name and attribs

scope = iterable to use for a frame; preferably a hashable (dictionary).
caseMatches = regex to match function names against; blank matches every TestCase subclass
caseSuperCls = superclass of test cases; unittest.TestCase by default
caseSorter = sort test cases

A search for "python sort unit test order in method definition order" got me here, and the documentation isn't particularly useful either ("Function" it says, what is it passed, what is it expected to return?)

Reading the four questions linked to at the start of this one yields only three unique answers:

• Don't sort your test methods (not an answer).

• Put numbers in your test method names (ugly, and means you need to renumber them all if you want to put new tests in the middle, or use ever-growing strings, etc.).

• Go through all this rigmarole, which it looks like the poster had success with but isn't exactly easy to understand and digs into Python 3's internals for the function line number.

sortTestMethodsUsing expects a function like Python 2's cmp, which has no equivalent in Python 3 (I went to check if Python 3 had a ` spaceship operator yet, but apparently not; they expect you to rely on separate comparisons for < and ==, which seems much a backwards step...). The function takes two arguments to compare, and must return a negative number if the first is smaller. Notably in this particular case, the function may assume that the arguments are never equal, as unittest will not put duplicates in its list of test names.

With this in mind, here's the simplest way I found to do it, assuming you only use one TestCase class:

def make_orderer():
    order = {}

    def ordered(f):
        order[f.__name__] = len(order)
        return f

    def compare(a, b):
        return [1, -1][order[a] < order[b]]

    return ordered, compare

ordered, compare = make_orderer()
unittest.defaultTestLoader.sortTestMethodsUsing = compare

Then, annotate each test method with @ordered:

class TestMyClass(unittest.TestCase):
    @ordered
    def test_run_me_first(self):
        pass

    @ordered
    def test_do_this_second(self):
        pass

    @ordered
    def test_the_final_bits(self):
        pass

if __name__ == '__main__':
    unittest.main()

This relies on Python calling annotations in the order the annotated functions appear in the file. As far as I know, this is intended, and I'd be surprised if it changed, but I don't actually know if it's guaranteed behavior. I think this solution will even work in Python 2 as well, for those who are unfortunately stuck with it, though I haven't had a chance to test this.

If you have multiple TestCase classes, you'll need to run ordered, compare = make_orderer() once per class before the class definition, though how this can be used with sortTestMethodsUsing` will be more tricky and I haven't yet been able to test this either.

For the record, the code I am testing does not rely on the test order being fixed - and I fully understand that you shouldn't rely on test order, and this is the reason people use to avoid answering this question. The order of my tests could be randomised and it'd work just as well. However, there is one very good reason I'd like the order to be fixed to the order they're defined in the file: it makes it so much easier to see at a glance which tests failed.

StackExchange Code Review Q#122532, answer score: 11