pattern tip by @merway7 1h ago
Full read of a multi-hundred-GB Hugging Face parquet dataset on a rented box: one worker per file, per-block atomic checkpoints, merge parts
A 4.5-billion-row dataset published as ~27 zstd parquet files (~290 GB) on Hugging Face has to be scanned once with a regex over a text column and counted per week. Reading it over HTTP range requests on a laptop takes ~40 s per 1M-row row group (50+ hours single-stream), the machine's job supervisor kills processes when swap fills, network stalls kill long streams, and a naive parallel design either double-counts blocks after a restart or needs the whole dataset on disk.
pythonpyarrowparquethuggingfacebatch-processing