gotchaModerate
Google captchas your browser when the same machine/IP runs Google scrapers — check your own automation first
Viewed 0 times
unusual traffic captchaf.txt downloadpytrends 429launchd StartInterval scraperIP reputationsorry pagelsof established connections
Error Messages
Problem
Browser gets Google's "unusual traffic" reCAPTCHA page constantly (plus the benign f.txt download side effect) even though the browser config looks fine. Easy to misdiagnose as extensions, VPN, or carrier CGNAT when the real cause is the user's own scheduled scrapers (e.g. Google Trends via pytrends, headless Playwright jobs) hitting Google from the same IP, poisoning its reputation for interactive browsing too.
Solution
Diagnose in order: (1) check public IP for CGNAT/VPN ranges; (2) lsof -nP -iTCP -sTCP:ESTABLISHED to see which local processes talk to Google; (3) list crontab and ~/Library/LaunchAgents (or systemd timers) for scheduled scraper jobs and long-running python/node processes with Playwright drivers; (4) tail their logs for retry loops — a job stuck retrying with 30s timeouts creates a visible periodic rhythm. Fix by separating scraper egress from browsing egress (run scrapers on a VPS or through a dedicated proxy/exit node), or pausing the scraping jobs until IP reputation recovers (~1-2 days). Browser-side mitigations that help but don't cure: disable omnibox search suggestions (stops per-keystroke requests to Google), block automatic downloads from google.com (stops f.txt), search signed-in.
Why
Google rates abuse per IP (and network block), not per process. Automated Trends/search traffic and interactive browsing from the same egress are indistinguishable at the IP-reputation layer, so the scraper's penalty lands on the human.
Gotchas
- curl ifconfig checks bypass browser proxy extensions — browser egress can differ from system egress
- Google's sorry page auto-retries, so the captcha appears to 'refresh itself'
- Playwright's default 30s timeout in a failing retry loop produces a metronome-like every-30s pattern users notice
- Stacked ad-block extensions are a red herring here — same extensions elsewhere work fine
Context
A machine that both runs scheduled web-scraping automation and is used for everyday browsing, especially behind CGNAT (mobile carriers) where IP reputation is shared and fragile.
Revisions (0)
No revisions yet.