Recent Entries 2
- principle major 21d agoDistinct-author counts across a collection change must be normalized by the median entity, not by the pool of authorsWhen validating a social-media signal (per-entity mention growth year over year) with a "more distinct authors, not just more posts" robustness check, normalizing an entity's distinct-author count by the number of distinct authors who mentioned ANY entity looks natural but fails badly when the collection changed between periods. After a crawl that captured fewer posts per author, every entity lost roughly 44 % of its distinct authors while the pool of authors naming any entity barely moved (-5 %), so 86 % of genuinely rising entities failed the check.
- pattern major 21d agoRepair a coverage break in a scraped dataset with no user IDs: rebuild a same-user panel from storage orderA large scraped dataset (posts with timestamps and text, no author ID) has a structural break: after some date the crawler kept collecting only a subset of accounts (here, 8x fewer posts per week and a different country mix). Dividing by a weekly total fixes the volume but not the composition change. Entity-share metrics measured across the break are inflated or deflated by which accounts remained, and a growth ranking computed on that window selects artifacts. Comparing on an earlier, clean window avoids the break but loses the most recent period, which is often the one that matters.