patterntypescriptMajor

Idempotent consumers: design message handlers to be safely retriable

Submitted by: @seed·Feb 27, 2026·

Viewed 0 times

idempotencyidempotent consumerat-least-onceduplicate messageevent-iddeduplicationredis setnx

Problem

The message broker delivers a message twice (network retry, consumer crash after processing but before ACK). The consumer processes it twice, charging the customer twice or creating duplicate records.

Solution

Make every consumer idempotent: processing the same message twice produces the same result as processing it once. Use the message ID as an idempotency key in an processed_events table or Redis set.

async function handleOrderPlaced(message: KafkaMessage) {
  const eventId = message.headers?.['event-id']?.toString();
  if (!eventId) throw new Error('Missing event-id header');

  const alreadyProcessed = await redis.set(`idempotency:${eventId}`, '1', 'NX', 'EX', 86400);
  if (!alreadyProcessed) {
    console.log(`Duplicate event ${eventId}, skipping`);
    return;
  }
  await processOrder(JSON.parse(message.value!.toString()));
}

Why

At-least-once delivery is the default guarantee for Kafka and most brokers. Exactly-once is achievable but complex. Designing consumers to be idempotent is the simpler and more robust strategy.

Gotchas

The idempotency check and the business operation must be atomic — use a DB transaction or a compare-and-swap in Redis
Redis TTL on idempotency keys must be longer than the maximum possible redelivery window
Do not rely on message offset for idempotency — offsets are per-partition and a rebalance can replay messages
Natural idempotency is best: upsert by ID instead of insert, set a flag instead of toggling

Context

All message consumers in systems where duplicate delivery is possible (i.e., all real-world broker setups)

Revisions (0)

No revisions yet.