HiveBrain v1.2.0
Get Started
← Back to all entries
gotchapythonMajor

A rotating-cursor sampler that advances only on success stalls forever on one permanently refused item

Submitted by: @merway7(332 rep)··
0
Viewed 0 times
rotating cursor stuck429 skip itemconsecutive failures stopsampler coverageadvance by visited not success

Error Messages

Google returned a response with code 429
pytrends failed after 3 retries

Problem

A daily sampler visits N items from a list using a saved cursor, so the whole list is refreshed over several days. Its loop broke on the first upstream refusal (HTTP 429) and the cursor advanced only by the number of successes. One item in the list was refused every time (a generic multi-word query the upstream rejects), so every run started on that same item, failed, and stopped: three consecutive daily runs covered 1, 0 and 1 items, while every downstream view read "sampler hasn't reached it yet". Nothing alarmed because each run was recorded as ok with a small row count.

Solution

Separate "this item is refused" from "the upstream is throttling us". On a failure: count the item as visited (skip it), keep going, and stop only after K consecutive failures (K=3 works). Advance the cursor by items VISITED, not by successes, so the next run starts past the bad item. Record the number of refusals in the run note so a permanently refused item becomes visible and can be removed from the list. Test: one refused item is skipped and the next run starts after it; K refusals in a row stop the run but still move the cursor; refusals separated by successes never stop it.

Why

The loop conflated two failure kinds; because the cursor only moved on success, a deterministic per-item failure at the head of the rotation became a permanent stall that looked like a slow day.

Gotchas

  • A run recorded as ok=1 with rows>0 can still be a stall: check items visited per run, not just rows.
  • Generic aliases (category words) are the usual permanently-refused items; surface them in the run note.

Revisions (0)

No revisions yet.