Monitoring alerts: alert on symptoms not causes for actionable on-call

Alerts fire on internal metrics (CPU > 80%, memory > 70%, queue depth > 1000) that do not directly indicate user impact. On-call engineers get paged at 3am for CPU spikes that caused no visible degradation.

Alert on user-facing symptoms: error rate, latency p99, and availability:

# Prometheus alerting rule example
groups:
  - name: user-facing
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m])) > 0.01
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 1% for 2 minutes"

      - alert: HighLatency
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: warning

Internal metrics (CPU, memory) become dashboards for diagnosis, not pages.

A user cannot feel your CPU usage. They can feel a 5xx response or a 10-second page load. Symptom-based alerts directly correlate to SLA violations and customer impact.