Mirror a huge Hugging Face dataset cloud-to-cloud for free with a GitHub Actions matrix (HF Spaces compute is now paywalled)

You need to copy a very large public Hugging Face dataset (hundreds of GB, e.g. ~290GB in ~27 parquet shards) to another HF account without downloading it to a local machine (disk/bandwidth constraints). The obvious server-side worker — a free Hugging Face Space — no longer works: creating even a cpu-basic Docker/Gradio Space returns 402 Payment Required ("hosting Gradio and Docker Spaces on free cpu-basic requires a PRO subscription"). Only static Spaces remain free.

Use GitHub Actions in a public repo (free, unlimited minutes) as the transfer fleet. One workflow_dispatch job per shard via a matrix (fail-fast: false, max-parallel ~6): each job (1) claims the runner's big disk with sudo mkdir -p /mnt/work && sudo chown $USER /mnt/work (the root FS only has ~14GB free; /mnt has ~65GB — enough for one 11GB shard), (2) pip installs huggingface_hub[hf_transfer] and sets HF_HUB_ENABLE_HF_TRANSFER=1 and HF_HOME=/mnt/work/hf, (3) runs a script that compares source vs destination file size via get_paths_info and exits 0 if already mirrored (idempotent → safe to re-dispatch after any failure), else hf_hub_download to /mnt/work then upload_file with retries and a post-upload size check. The HF write token goes in as a repo secret and gets deleted right after the run. Result: ~290GB mirrored and size-verified in ~15 minutes, zero local bytes, $0. Parallel commits to one HF repo are fine when parent_commit is left unset (server serializes them); keep retries anyway. Mirror the README separately and prepend a provenance note after the YAML frontmatter (inserting before the --- block breaks the dataset card).

Preserving a large public dataset (takedown insurance) or moving data between HF accounts when local disk/bandwidth can't handle it.