patternjavascriptModerate
Retry Strategies with Exponential Backoff in BullMQ
Viewed 0 times
retrybackoffexponential backofftransient errorbullmq attemptsjob retry
Problem
A job fails due to a transient error (network timeout, rate limit, temporary service outage). Retrying immediately hammers the failing dependency and likely fails again.
Solution
Configure BullMQ's attempts and backoff options. Use exponential backoff so retries are spaced increasingly apart (1s, 2s, 4s, 8s...). Add jitter to avoid synchronized retry storms from many concurrent jobs.
Why
Exponential backoff gives the failing dependency time to recover. It also reduces load during an outage, which helps the system stabilize faster.
Gotchas
- BullMQ's built-in exponential backoff doubles the delay each attempt. For jitter, implement a custom backoff function.
- Jobs that have exhausted all attempts move to the 'failed' state, not the dead-letter queue. You must manually handle the failed state.
- Non-transient errors (invalid data, programming bugs) should not be retried. Detect them in the worker and throw a non-retriable error, or set attempts: 1.
Code Snippets
BullMQ job with exponential backoff
await queue.add('processPayment', data, {
attempts: 5,
backoff: {
type: 'exponential',
delay: 1000, // starts at 1s, then 2s, 4s, 8s, 16s
},
});
// Custom backoff with jitter:
// backoff: { type: 'custom' }
// Then in worker options:
// settings: { backoffStrategy: (attempt) => Math.min(1000 * 2**attempt + Math.random()*1000, 30000) }Revisions (0)
No revisions yet.