principletypescriptModerate
Smoke tests vs regression tests: run different test sets at different trigger points
Viewed 0 times
smoke testsregression testsci speedpost-deploytest taggingtest pipeline
Problem
Running the full regression suite on every commit takes 45 minutes. Developers skip tests locally, CI queues back up, and the feedback loop from code change to test result is too slow to be useful.
Solution
Distinguish smoke tests (5-10 critical path tests that prove the system is alive) from regression tests (the full suite that validates all behavior). Run smoke tests on every commit. Run regression tests on PR merge to main or nightly. Run smoke tests again after every deployment as a post-deploy gate.
Why
Smoke tests fail fast on obvious breakage without the full suite cost. Regression tests provide comprehensive coverage at lower frequency where the latency is acceptable. Post-deploy smoke tests catch environment-specific issues that CI does not surface.
Gotchas
- Smoke tests must be independent and idempotent — they run against live environments and must not create lasting side effects
- Keep smoke test count low — if 'smoke' expands to 50 tests it becomes a mini-regression suite and loses its speed advantage
- Tag smoke tests with @smoke or a --grep filter so they can be run in isolation
- A failing smoke test post-deploy should trigger an automatic rollback in a mature CD pipeline
Revisions (0)
No revisions yet.