Alert fatigue — designing actionable alerts that don't get ignored

Too many alerts fire constantly. Team members ignore alerts because most are false positives or non-actionable. Critical alerts get lost in the noise. On-call engineers are burned out.

(1) Every alert must have a runbook: what it means, impact, and exact steps to fix. No runbook = delete the alert. (2) Alert on symptoms, not causes: alert on error rate > 5%%, not on CPU > 80%% (high CPU might be fine). (3) Use severity levels: page (wake someone up) only for customer-impacting issues. Everything else is a ticket/notification. (4) Set appropriate thresholds: alert on sustained anomalies (5 min), not spikes (30 sec). Use percentage-based thresholds, not absolute numbers. (5) Aggregate: one alert for 'service degraded' not 100 alerts for individual instances. (6) Auto-resolve: alerts must clear automatically when the condition resolves. (7) Review: weekly alert review — delete noisy alerts, tune thresholds, merge duplicates.

Alert fatigue is a real danger — when everything alerts, nothing alerts. Teams learn to ignore notifications, and the one critical alert that matters gets missed among hundreds of irrelevant ones.