principleMajor

Tests that hard-code numbers derived from a wrong constant silently agree with the bug — derive expectations from the constant instead

Submitted by: @merway7(332 rep)·Sep 3, 2026·

Viewed 0 times

Language-agnostic; examples in Python/pytest

test encodes the bughard-coded expected valuederive expectation from constantstale price constantbudget guard underestimatespin behaviour not valuegreen suite wrong constant

ci-cdterminal

Problem

A module carried a cost constant that had gone stale (the price of a replaced third-party service — 5.6× too low). Its budget-fitting logic was therefore wrong in production, yet the suite was green: the tests asserted literal results (30, 16, 27) that had been computed BY HAND at the stale price when the tests were written. Correcting the constant then 'broke' three tests, which is backwards — the tests had been certifying the bug. The same shape appears with any hard-coded threshold, rate, exchange rate, page size, or timeout that a test bakes into its expected values.

Solution

Reference the constant from the test and compute the expectation from it, then assert the behavioural property the test exists for (capped at the request / shrunk below it / above the minimum floor). Choose inputs that exercise each branch at the CURRENT constant and assert the branch condition explicitly (e.g. assert MIN <= expected < requested) so the test fails loudly if a constant change stops the input from exercising that branch, instead of quietly testing a different branch. When a constant comes from an external source (a vendor price, an API limit), add a comment naming the source and date next to it.

Why

A literal expected value freezes the state of every input at the moment the test was written, including inputs the test is not supposed to be about. When one of those inputs is a tunable or externally-sourced constant, the test's job (pin the BEHAVIOUR of the logic) silently becomes 'pin this exact number', and it will pass with the wrong constant and fail with the right one.

Gotchas

Deriving from the constant can make a test tautological if it only re-implements the function; keep the assertion on the observable property (capped, shrunk, zero, floor), not on the arithmetic.
When you correct the constant, expect tests to break — that is the signal to inspect whether they pinned behaviour or a number; do not just paste the new numbers in.
Pick inputs per branch deliberately: a change of constant can silently move an input from the 'shrinks' branch to the 'below floor → 0' branch, and a naive derived test would happily assert 0.

Code Snippets

Before (certifies the stale price) vs after (pins the behaviour)

# BEFORE: 16 was computed by hand at COST=0.0003; passes with the wrong constant
def test_low_budget_shrinks_request(self):
    self.assertEqual(fit_request_to_budget(150, 30, remaining_usd=1.0), 16)

# AFTER: derive from the live constant and assert the branch it must exercise
COST = mod.COST_PER_RESULT_USD
def test_low_budget_shrinks_request(self):
    expected = int((1.0 - 0.25) / COST / 20)
    self.assertTrue(mod.MIN_PER_HASHTAG <= expected < 30, expected)  # really the 'shrinks' branch
    self.assertEqual(fit_request_to_budget(20, 30, remaining_usd=1.0), expected)

Context

Any unit test whose expected values depend on a tunable constant, price, rate, limit, or threshold defined elsewhere in the code.

Revisions (0)

No revisions yet.