Full read of a multi-hundred-GB Hugging Face parquet dataset on a rented box: one worker per file, per-block atomic checkpoints, merge parts

A 4.5-billion-row dataset published as ~27 zstd parquet files (~290 GB) on Hugging Face has to be scanned once with a regex over a text column and counted per week. Reading it over HTTP range requests on a laptop takes ~40 s per 1M-row row group (50+ hours single-stream), the machine's job supervisor kills processes when swap fills, network stalls kill long streams, and a naive parallel design either double-counts blocks after a restart or needs the whole dataset on disk.

Split by file, not by row: a driver spawns one worker process per parquet file, N at a time (N = cores; ~0.8 GB RSS each with pyarrow ParquetFile.iter_batches(batch_size=50_000, row_groups=[rg], columns=[only what you need])). Each worker downloads its file with huggingface_hub.hf_hub_download(repo_id, filename, repo_type='dataset', local_dir=...) (resumable), reads it from disk, and deletes it when done, so disk use is N x one file, not the whole dataset. State is written after every completed row group as a JSON checkpoint via tmp file + os.replace (atomic), and the worker is started with --resume, so a crash mid-block loses only that block's in-memory partial and never double counts; the driver restarts a failed worker up to K times with a pause. Each checkpoint records the number of row groups per file (from the parquet footer) so the merge step can refuse to write a partial result. Merge sums dict counters across parts, re-sorts and caps top-N receipts, concatenates per-row 'focus' records, and preserves the exact schema of the single-process sampler so every downstream script reads the full read unchanged. The driver prints blocks done / expected, rows, blocks per minute and an ETA every minute by re-reading the part files. Provide a --parquet-dir option that reads local *.parquet files instead of the hub so the whole chain (run -> merge -> downstream) is testable end to end on two tiny parquet files written with pyarrow in the test.

Parquet files are independent and row groups are the natural unit of work and of memory; a checkpoint written only after a whole row group makes restarts idempotent without any bookkeeping of partial batches, and per-file workers keep the merge trivial because every part has the same schema as a single-process run.

Turning a 2% sample into an exact count once the owner wanted six-sigma certainty; the same design works for any one-off scan of a large hub dataset on a rented 8-32 vCPU machine for about a dollar.