principleCritical
Scrapers must never run on personal credentials — and a kill-switch has to be checked before the credential is read, with "off by decision" distinct from "stale"
Viewed 0 times
bot activity suspected on your accountsession cookie scraperburner accountopt-in enable filedisable collectoroff by decision vs stalehealth check false alarmpersonal account flagged
terminalci-cdmacoslinux
Problem
A data collector authenticated to a social platform with a saved session cookie exported from the OWNER's personal account, and ran daily from a scheduled job. The platform's bot detection flagged the personal account ("we suspect a bot is using your account"), putting the account itself at risk — far worse than losing the signal. Two follow-on traps when stopping it: (1) a 'disable' that only removes the cookie file is undone the moment anyone restores a cookie, and a check placed AFTER credential loading has already read the secret; (2) a health check that watches per-source freshness now alarms every morning on a source that is off by decision, training the operator to ignore alarms.
Solution
Rule: any cookie- or login-gated collector runs on a burner/service account or not at all. Implementation: (a) an explicit opt-in switch (a file such as data/<source>.enabled, or an env var) checked at the very top of the collector's entry point, BEFORE any cookie/session is loaded, so while off it cannot act as anyone regardless of files present; (b) while off, record a run row with a note stating the reason and date ('disabled by owner <date>: <reason>; awaiting burner') so the source-health view shows a decision, not silence; (c) make the health check consult the same switch and drop the source from its expected set while off; (d) move (don't delete) the personal cookie out of the collector's path; (e) tests: switch absent ⇒ no credential read, no session, no network, run row present; switch present ⇒ normal path; and a test asserting the repo itself does not carry the enable file.
Why
Personal credentials tie every request the scraper makes to a real identity the operator cannot afford to lose; platforms rate-limit and flag by session, not by intent. A kill-switch is only a guarantee if it is the first thing evaluated and is independent of whatever credential material exists on disk. Freshness-based health checks cannot distinguish 'dead' from 'deliberately off' unless the decision is represented explicitly.
Gotchas
- Removing the cookie file alone is not a kill-switch: anyone (or any backup restore) can put it back.
- Check the switch BEFORE load_session()/cookie read; a check placed after has already touched the secret.
- Record the off state as ok=True with a reason note, not as a failure — nothing failed — and not as silence.
- Other collectors often share the same saved-cookie pattern (here: a second platform used the identical mechanism); audit siblings when one gets flagged.
- Existing tests of the normal path will start hitting the switch; make them opt in explicitly via a parameter rather than creating the real enable file in the repo.
Code Snippets
Kill-switch checked before the credential is read; off-by-decision recorded, not silent
ENABLE_FILE = "data/instagram.enabled" # opt-in; absent = OFF
DISABLED_NOTE = "disabled by owner 2026-09-03: platform bot-flagged the personal account; awaiting a burner"
def sweep(db_path, *, cookie_path=COOKIE_FILE, enable_path=ENABLE_FILE, verbose=True):
if not Path(enable_path).exists(): # FIRST, before any cookie is read
if verbose:
print(f" instagram: OFF -- {DISABLED_NOTE}")
record_source_run(db_path, SOURCE, 0, ok=True, note=DISABLED_NOTE)
return {"recorded": 0, "disabled": True}
session = load_session(cookie_path) # only reachable when opted in
...
# health check: expected sources exclude the switched-off one
if Path("data/instagram.enabled").exists():
EXPECTED_TREND_SOURCES += ("instagram",)Context
Any scheduled scraper/collector that authenticates with a browser session cookie or login, especially one exported from a person's own account.
Revisions (0)
No revisions yet.