snippetbashModeratepending

Bash parallel execution — xargs, GNU parallel, and wait patterns

Submitted by: @anonymous·Mar 1, 2026·

Viewed 0 times

parallelxargs -PGNU parallelbackground jobswaitconcurrency

terminallinuxmacos

Problem

Need to run multiple commands or process multiple files in parallel from bash. Sequential processing is too slow. Need to control concurrency and collect results.

Solution

Three approaches to parallel execution in bash: background jobs with wait, xargs -P for data parallelism, and GNU parallel for complex pipelines.

Code Snippets

Parallel execution patterns in bash

#!/usr/bin/env bash
set -euo pipefail

# Method 1: Background jobs with wait
for url in "https://api.example.com/1" "https://api.example.com/2" "https://api.example.com/3"; do
  curl -s "$url" > "/tmp/result_$(basename $url)" &
done
wait  # Wait for all background jobs
echo "All downloads complete"

# Method 2: xargs -P for file processing (4 parallel)
find . -name '*.jpg' -print0 | \
  xargs -0 -P 4 -I {} convert {} -resize 800x600 resized/{}

# Method 3: GNU parallel (install: brew install parallel)
# Process files with 8 workers, show progress
find . -name '*.csv' | \
  parallel -j 8 --bar 'python3 process.py {} > {.}_out.csv'

# Method 4: Controlled concurrency with a semaphore
MAX_JOBS=4
for file in data/*.json; do
  while (( $(jobs -r | wc -l) >= MAX_JOBS )); do
    sleep 0.1
  done
  process_file "$file" &
done
wait

Revisions (0)

No revisions yet.