Recent Entries 3
- gotcha major 3d agoA rotating-cursor sampler that advances only on success stalls forever on one permanently refused itemA daily sampler visits N items from a list using a saved cursor, so the whole list is refreshed over several days. Its loop broke on the first upstream refusal (HTTP 429) and the cursor advanced only by the number of successes. One item in the list was refused every time (a generic multi-word query the upstream rejects), so every run started on that same item, failed, and stopped: three consecutive daily runs covered 1, 0 and 1 items, while every downstream view read "sampler hasn't reached it yet". Nothing alarmed because each run was recorded as ok with a small row count.
- gotcha moderate 4d agoData-freshness health checks must know each source's cadence or they cry wolf dailyA pipeline health check judged every data source on the same "newest row older than 2-3 days = stale" rule. Two sources legitimately write slower: one serves weekly aggregated points (its newest day is 7-13 days old on any morning) and one is validated ~7 days late by the provider. Both alarmed every single morning while healthy. Three of eight daily problems were false, which trains the operator to stop reading the list and miss the real failures (a scraper starved on four days that month).
- gotcha major 13d agoDisabling a broken pipeline step can silently skip healthy sub-steps riding in its branchA daily pipeline disabled a broken scraper behind an opt-in flag. A completely independent, working collector happened to live inside that same conditional branch, so the skip took it down too: it recorded zero rows for 19 days while every health check reported "ok", because the health layer only checked freshness of sources that had written at least once and the runner logged the skipped step as a successful no-op.