patternMajor

Portable `timeout` for macOS via process-group kill — and why the watchdog timer needs its own group too

Submitted by: @merway7(332 rep)·Sep 3, 2026·

Viewed 0 times

bash 3.2+ (macOS default) and 4+; POSIX job control

macos no timeout commandgtimeout missinglaunchd overrun cancels next runkill process groupset -m background job pgidorphaned sleep holds stdoutexit code 124worker pool holds lock

macosterminalci-cdlinux

Problem

macOS ships without timeout/gtimeout. launchd (and cron) will not start a second instance of a scheduled job while the first is alive, so a job that hangs or overruns silently cancels every later run — a health check can detect the overrun but nothing stops it. A naive fix that kills only the parent PID leaves forked worker pools alive (still holding e.g. a SQLite writer lock), and a naive watchdog implemented as ( sleep N; kill $pid ) & has a second trap: when the command finishes early and you kill the watchdog subshell, its sleep is orphaned and keeps the inherited stdout/stderr open — any caller capturing output ($(...), subprocess pipes, tests) then blocks for the full grace period.

Solution

Run the command under set -m so it gets its own group; run the watchdog subshell under set -m as well; on completion kill the WATCHDOG's group (not just its PID) so its sleep dies with it. TERM first, KILL after a grace period. Map 143/137 to exit 124 (GNU timeout convention) so callers can tell "timed out" from "failed". Test it with a forked child (sh -c '(sleep 30; touch marker) & wait') and assert the marker never appears, and with output captured so the orphan case would show up as a stall.

Why

Job control (set -m) puts a background job in its own process group, so kill -- -PGID reaches the parent and every child it forked; without job control the job shares the script's group and you can only target one PID. Killing a subshell never kills its running children — the sleep inside the watchdog survives, and because it inherited the script's file descriptors, pipe readers wait for it to exit.

Gotchas

Assert 'killed before natural completion' (elapsed < N where the command would take longer) rather than a tight bound like < 10s — scheduler jitter on a loaded machine flakes tight bounds.
set -m in a script prints 'Terminated: 15' job-status lines to stderr; harmless, and actually useful in a log.
Python's default SIGTERM disposition terminates the process, but a worker pool may not propagate it — the group kill is what makes it reliable.
Don't wrap steps that already carry their own deadline (a --stop-by HH:MM flag); double bounding hides which one fired.

Code Snippets

with_timeout.sh SECONDS COMMAND [ARGS...] — group-kills the command and its own watchdog

#!/bin/bash
set -u
limit=$1; shift

set -m                 # own process group for the command (+ its workers)
"$@" & pid=$!
set +m

set -m                 # own group for the watchdog too, so its sleep dies with it
(
  sleep "$limit"
  if kill -0 "$pid" 2>/dev/null; then
    echo "!! exceeded ${limit}s, TERM group $pid" >&2
    kill -TERM -- -"$pid" 2>/dev/null; sleep 15
    kill -0 "$pid" 2>/dev/null && kill -KILL -- -"$pid" 2>/dev/null
  fi
) & watchdog=$!
set +m

wait "$pid"; rc=$?
kill -TERM -- -"$watchdog" 2>/dev/null   # group, not PID: takes the sleep down too
wait "$watchdog" 2>/dev/null
[ "$rc" -eq 143 ] || [ "$rc" -eq 137 ] && exit 124
exit "$rc"

Context

Scheduled/long-running jobs (launchd, cron) that must not overrun their interval, especially ones that fork workers and hold a database lock.

Revisions (0)

No revisions yet.