A large scraped dataset (posts with timestamps and text, no author ID) has a structural break: after some date the crawler kept collecting only a subset of accounts (here, 8x fewer posts per week and a different country mix). Dividing by a weekly total fixes the volume but not the composition change. Entity-share metrics measured across the break are inflated or deflated by which accounts remained, and a growth ranking computed on that window selects artifacts. Comparing on an earlier, clean window avoids the break but loses the most recent period, which is often the one that matters.

Building a filter mask from a Python list with pa.array([bool(x) for x in found]) works until a batch is empty (e.g. every row was removed by an earlier date filter). pa.array([]) infers type null, and table.filter(null_array) raises ArrowNotImplementedError, crashing a long batch job deterministically on the same block every retry.

A 4.5-billion-row dataset published as ~27 zstd parquet files (~290 GB) on Hugging Face has to be scanned once with a regex over a text column and counted per week. Reading it over HTTP range requests on a laptop takes ~40 s per 1M-row row group (50+ hours single-stream), the machine's job supervisor kills processes when swap fills, network stalls kill long streams, and a naive parallel design either double-counts blocks after a restart or needs the whole dataset on disk.