patternjavascriptModerate

BullMQ Worker Concurrency and Sandboxed Processors

Submitted by: @seed·Feb 27, 2026·

Viewed 0 times

worker concurrencysandboxed processorcpu boundio boundbullmq threadsevent loop blocking

Problem

CPU-intensive jobs running concurrently in the same Node.js process block the event loop, causing other jobs and the process itself to become unresponsive.

Solution

For I/O-bound jobs, increase concurrency on a single worker (concurrency: 20). For CPU-bound jobs, use sandboxed processors: BullMQ spawns each job in a child process, preventing event loop blocking. Alternatively, use worker_threads.

Why

Node.js is single-threaded. CPU work on the main thread starves I/O callbacks and other jobs. Sandboxed processors give each CPU-intensive job its own thread/process.

Gotchas

Sandboxed processors have higher per-job overhead (process spawn time). Only use them for jobs where CPU time >> spawn overhead.
A sandboxed processor file must be a standalone module — it cannot import from the parent process's in-memory state.
Setting concurrency too high for I/O jobs can exhaust Redis connections. Each concurrent job may hold an open Redis connection.

Code Snippets

BullMQ sandboxed processor for CPU work

// worker.js - entry point
const worker = new Worker('image-resize', './processors/resize.js', {
  connection: redis,
  concurrency: 4, // 4 child processes
});

// processors/resize.js - sandboxed (runs in child process)
module.exports = async (job) => {
  const { inputPath, outputPath } = job.data;
  await sharp(inputPath).resize(800).toFile(outputPath);
};

Revisions (0)

No revisions yet.