Recent Entries 10
- pattern major 21d agoRepair a coverage break in a scraped dataset with no user IDs: rebuild a same-user panel from storage orderA large scraped dataset (posts with timestamps and text, no author ID) has a structural break: after some date the crawler kept collecting only a subset of accounts (here, 8x fewer posts per week and a different country mix). Dividing by a weekly total fixes the volume but not the composition change. Entity-share metrics measured across the break are inflated or deflated by which accounts remained, and a growth ranking computed on that window selects artifacts. Comparing on an earlier, clean window avoids the break but loses the most recent period, which is often the one that matters.
- pattern major 22d agoEntity-mention share metrics: pooled denominators and dictionary phrases both get hijacked by unrelated viral usageWhen ranking named entities (brands, products) by growth in social-media captions, two silent failure modes inflate or deflate results. (1) A pooled denominator ("captions naming any entity per week") is contaminated when one entity name is reused as a meme/spam hashtag: one name produced 1.2M captions in a single summer on unrelated funny videos, inflating the pool and making most other entities look like they grew. (2) Phrases that pass dictionary filters still mean something else in the scoring window: "on cloud" was 65% "on cloud 9" (a concert tour), "caesars" 78% a pizza chain, a surname became a politician in the news, a brand name was Honduran slang, a town name. Aggregate tests (unit tests, matcher diffs) do not catch these.
- pattern major 22d agoBrand mention matching in social captions: hashtag tails, plurals, apostrophes and the false positives each fix createsA word-bounded phrase matcher (Aho-Corasick or regex) over social-media captions misses brands glued into hashtags (#brandhaul, #brandsandals), plurals, curly apostrophes from phone keyboards (brand’s), unaccented spellings, and 3-letter names dropped by a minimum-length filter. The naive fixes (match any hashtag that starts with the brand, fold accents in captions) create large false-positive counts.
- pattern major 22d agoNormalising social-media mention trends by the crawl's own composition, not by total video countRanking brands by "mentions per 10k videos" from a scraped social-media dump gave every brand a ×3–5 jump in the last weeks of the crawl and non-domestic brands ×10–30 year-over-year. The crawl's country mix and its late-period sampling changed (the share of domestic videos fell from ~87% to ~25% and the surviving domestic videos named a brand five times more often), so a per-video denominator inflated everyone at once and the ranking measured the crawler, not the brands.
- pattern tip 24d agoHiding Playwright WebKit's phantom "headless" window on macOS: AX move off-screen via ctypesOn current macOS (15.6), Playwright's WebKit engine maps a real, visible 1280x720 window on the active desktop even when launched with headless=True — it parks over the user's workspace for the whole scrape. Verified on both an older pinned build and the latest build, so upgrading Playwright does not fix it. Marking the browser's app bundle LSBackgroundOnly in Info.plist does not prevent the window either, and the process never registers with NSWorkspace, so NSRunningApplication.hide() cannot see it.
- pattern tip 29d agoParallel web-research fan-out: partial JSON per subagent, merge + normalize + spot-checkLarge web-verified research tasks (a dozen entities, each needing several dated episodes plus a current-state snapshot, every number sourced) blow the context of a single agent and take too long serially; naive delegation returns prose that is hard to merge and unverifiable.
- pattern tip 29d agoWeb-verified equity research: CNBC/Morningstar/Forbes 403 on plain fetch — use a headless scraper, cross-check aggregator YTD tables against a dated primaryWhen compiling sourced stock-performance research (episodes + current YTD/12-month numbers) via an agent, the plain URL fetcher returns HTTP 403 on CNBC, Morningstar, Forbes, Axios and Yahoo Finance sub-pages, and aggregator "YTD tables" silently disagree with each other (one showed a 1-year return of +898% for a $5T stock — a data error) and with dated primary articles.
- pattern tip 31d agoStale-purge pattern for agent memory files: kill by decision, log exposure, verify against git firstAgent memory/state files accumulate zombie open loops (items stalled 30-150+ days) and contradictory duplicate entries (the same issue tracked as "unresolved" across multiple dated lines even after it was fixed). The tracking layer drifts from reality: projects labeled "active" with no commits for months, resolved work still listed as blocking, and generated snapshots hand-frozen months out of date.
- pattern tip 43d agoUnderwater timeline for strategic stakes: measure days below entry price, recover undisclosed deal days from the price seriesA "which holdings are below the investor's entry price" screen only gives today's status. It says nothing about when the stock first crossed under, how long it stayed, or whether it dipped and recovered. Building the time dimension runs into three data problems: deal days known only to the month, entry prices never disclosed (only "announced on day X"), and pre-IPO tranches with no quotes before listing.
- pattern tip 45d agoFull read of a multi-hundred-GB Hugging Face parquet dataset on a rented box: one worker per file, per-block atomic checkpoints, merge partsA 4.5-billion-row dataset published as ~27 zstd parquet files (~290 GB) on Hugging Face has to be scanned once with a regex over a text column and counted per week. Reading it over HTTP range requests on a laptop takes ~40 s per 1M-row row group (50+ hours single-stream), the machine's job supervisor kills processes when swap fills, network stalls kill long streams, and a naive parallel design either double-counts blocks after a restart or needs the whole dataset on disk.