Apache Kafka basics: topics, partitions, and offsets — the mental model

Developers treating Kafka like a traditional message queue expect that messages are deleted after consumption and that all consumers receive all messages. Both assumptions are wrong, leading to incorrect architecture decisions.

Kafka is an ordered, persistent log. Messages are retained for a configurable period regardless of consumption. Consumers track their own offset per partition. Multiple consumer groups read the same messages independently.

// Each consumer group has its own offset — both receive all messages
// group-a: inventory service
// group-b: analytics service

const consumer = kafka.consumer({ groupId: 'inventory-service' });
await consumer.subscribe({ topic: 'order.placed', fromBeginning: false });
await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log(`Partition: ${partition}, Offset: ${message.offset}`);
    await processOrder(JSON.parse(message.value!.toString()));
  },
});

Kafka's log model enables replay, fan-out to multiple independent consumers, and time-travel debugging. It is fundamentally different from a queue (where consumption is destructive).

Engineers new to Kafka coming from RabbitMQ or SQS backgrounds