Recent Entries 3
- gotcha moderate 4d agoData-freshness health checks must know each source's cadence or they cry wolf dailyA pipeline health check judged every data source on the same "newest row older than 2-3 days = stale" rule. Two sources legitimately write slower: one serves weekly aggregated points (its newest day is 7-13 days old on any morning) and one is validated ~7 days late by the provider. Both alarmed every single morning while healthy. Three of eight daily problems were false, which trains the operator to stop reading the list and miss the real failures (a scraper starved on four days that month).
- principle critical 4d agoScrapers must never run on personal credentials — and a kill-switch has to be checked before the credential is read, with "off by decision" distinct from "stale"A data collector authenticated to a social platform with a saved session cookie exported from the OWNER's personal account, and ran daily from a scheduled job. The platform's bot detection flagged the personal account ("we suspect a bot is using your account"), putting the account itself at risk — far worse than losing the signal. Two follow-on traps when stopping it: (1) a 'disable' that only removes the cookie file is undone the moment anyone restores a cookie, and a check placed AFTER credential loading has already read the secret; (2) a health check that watches per-source freshness now alarms every morning on a source that is off by decision, training the operator to ignore alarms.
- gotcha major 13d agoDisabling a broken pipeline step can silently skip healthy sub-steps riding in its branchA daily pipeline disabled a broken scraper behind an opt-in flag. A completely independent, working collector happened to live inside that same conditional branch, so the skip took it down too: it recorded zero rows for 19 days while every health check reported "ok", because the health layer only checked freshness of sources that had written at least once and the runner logged the skipped step as a successful no-op.