gotchapythonModerate
Data-freshness health checks must know each source's cadence or they cry wolf daily
Viewed 0 times
stale source false alarmweekly cadence freshnessmax age per sourcegoogle trends weekly pointsalarm fatigue
Error Messages
Problem
A pipeline health check judged every data source on the same "newest row older than 2-3 days = stale" rule. Two sources legitimately write slower: one serves weekly aggregated points (its newest day is 7-13 days old on any morning) and one is validated ~7 days late by the provider. Both alarmed every single morning while healthy. Three of eight daily problems were false, which trains the operator to stop reading the list and miss the real failures (a scraper starved on four days that month).
Solution
Keep a per-source allowance map (source -> max age in days) next to the default, and route every freshness rule (stale-by-age and no-rows-in-window) through one max_age_for(source, default) helper so the two rules cannot disagree. Measure the real cadence from the data before setting allowances: SELECT source, avg(gap between distinct days), max(gap) over the last 90 days; a source whose average gap is exactly 7.0 is weekly, not broken. Make the alarm message print the window it actually used. Daily sources keep the strict clock; test both that a weekly source 9 days old is silent and that it still alarms past its own window.
Why
Freshness thresholds were global while write cadences are per source; a weekly series can never satisfy a 3-day rule, so the check is structurally wrong for it rather than occasionally noisy.
Revisions (0)
No revisions yet.