principleMajorpending
Principle: Keep the build green
Viewed 0 times
green-buildCIflaky-testscontinuous-integrationbuild-health
Problem
Broken CI builds become normalized. Developers ignore test failures and merge despite red builds, leading to cascading quality issues.
Solution
Treat a broken build as the highest-priority issue:
- Never merge with a failing build
- Fix broken builds immediately (or revert)
- Don't add new work on top of broken builds
- The person who broke it fixes it (or reverts)
- Slow builds (>10 min) get ignored
- Target: < 5 minutes for feedback
- Use caching, parallelism, incremental builds
- Split slow tests into separate pipelines
- No flaky tests (fix or quarantine them)
- Pin dependencies (no random version changes)
- Use deterministic builds (same input = same output)
- Infrastructure failures should retry, not fail
- Slack/Teams notification for failures
- Build status badge in README
- Block merges when build is red
- Dashboard showing build health
- Build pass rate (target: > 99%)
- Time to fix broken builds (target: < 30 min)
- Build duration (target: < 10 min)
- Flaky test rate (target: < 1%)
- Don't delete it — fix the flakiness
- Quarantine: move to non-blocking pipeline
- Track flaky test rate as a metric
- Rules:
- Never merge with a failing build
- Fix broken builds immediately (or revert)
- Don't add new work on top of broken builds
- The person who broke it fixes it (or reverts)
- Make builds fast:
- Slow builds (>10 min) get ignored
- Target: < 5 minutes for feedback
- Use caching, parallelism, incremental builds
- Split slow tests into separate pipelines
- Make builds reliable:
- No flaky tests (fix or quarantine them)
- Pin dependencies (no random version changes)
- Use deterministic builds (same input = same output)
- Infrastructure failures should retry, not fail
- Make failures visible:
- Slack/Teams notification for failures
- Build status badge in README
- Block merges when build is red
- Dashboard showing build health
- Metrics to track:
- Build pass rate (target: > 99%)
- Time to fix broken builds (target: < 30 min)
- Build duration (target: < 10 min)
- Flaky test rate (target: < 1%)
- If a test is flaky:
- Don't delete it — fix the flakiness
- Quarantine: move to non-blocking pipeline
- Track flaky test rate as a metric
Why
A consistently green build is the foundation of continuous delivery. When broken builds are normal, nobody trusts the CI and quality degrades.
Revisions (0)
No revisions yet.