patternpythonMajor
Repair a coverage break in a scraped dataset with no user IDs: rebuild a same-user panel from storage order
Viewed 0 times
structural breakpanel constantsame-store comparisoncoverage changeno user idstorage order segmentationsurvey reweighting alternative
Problem
A large scraped dataset (posts with timestamps and text, no author ID) has a structural break: after some date the crawler kept collecting only a subset of accounts (here, 8x fewer posts per week and a different country mix). Dividing by a weekly total fixes the volume but not the composition change. Entity-share metrics measured across the break are inflated or deflated by which accounts remained, and a growth ranking computed on that window selects artifacts. Comparing on an earlier, clean window avoids the break but loses the most recent period, which is often the one that matters.
Solution
Check the dataset card and the storage order: many exports keep each account's posts contiguous and time-sorted. If consecutive rows move forward in time about 97% of the time, a step back in time marks a new account. Segment each row group (read its timestamp column first, because runs cross batch boundaries), then flag the runs that have posts in every quarter on both sides of the break. Requiring activity every quarter reduces false merges; a looser 'present in both periods' panel serves as a sensitivity check. Count entity mentions and the panel's own totals separately, and compare the recent period with the same period a year earlier within the panel, like same-store sales, with Poisson margins (se = sqrt(1/n1 + 1/n0)). Validate on a clean window: the panel's growth should track the whole sample's (r = 0.89 here across 1,172 entities, median ratio 0.96). Then label results by confidence: panel growth beyond its margin, panel growth relative to the median of its category peers, and a 'low confidence' fallback for entities too small for the panel, using a series spliced at the break.
Why
The break changes WHO is sampled, not just how many. Normalising by a total corrects scale only. A fixed panel holds composition constant, which is the standard fix in panel and survey statistics (constant panels, same-store comparisons). Chain-linking and category controls rest on stronger assumptions and are best kept as cross-checks.
Gotchas
- Merged accounts: when the next account starts after the previous one ends there is no step back; require continuous activity to limit contamination
- Runs cross batch boundaries: compute run flags on the whole row group before batching
- Selecting 'top risers' on a broken window picks artifacts (winner's curse); validate the panel on a clean window first
- Small entities have too few panel mentions; show them separately with a low-confidence label instead of mixing them into the ranking
- Sort default views by confidence first, or the noisiest small entities float to the top
Code Snippets
segment accounts at time steps back and flag panel rows
run = np.r_[0, np.cumsum(np.diff(ts) < 0)] # ts: seconds, storage order
has = {}
for k, (a, b) in WINDOWS.items():
h = np.zeros(run[-1] + 1, bool); h[run[(ts >= a) & (ts < b)]] = True; has[k] = h
panel_row = np.logical_and.reduce([has[k] for k in REQUIRED])[run]Context
Measuring year-over-year growth of entity mentions in a one-off scraped social dataset whose collection coverage collapsed in its final months.
Revisions (0)
No revisions yet.