TL;DR - Kafka is a partitioned, replicated commit log. Producers append, consumers read by offset at their own pace, and retention makes slow readers and replays free for the writer. Ordering holds per partition, durability comes from acks=all over three replicas; the design choices are the partition key, the partition count, and duplicate handling.
30-second version: "I would put an event log between the order service and its downstream consumers. Each order becomes a message keyed by order ID, so one order's events stay in sequence on one partition. Each downstream team runs its own consumer group at its own speed, and checkout latency becomes a single append acknowledged by two of three replicas. Consumers commit after processing and handle duplicates idempotently."
Why One Slow Consumer Stalls Checkout
An order service confirms a purchase and then calls five services in a row: inventory, ledger, email, analytics, fraud scoring. Each call is synchronous, so checkout waits for the slowest. On a normal day that is 5 x 40 ms = 200 ms. The day analytics runs a migration and its p99 climbs to 4 seconds, every checkout waits 4 seconds and the order service times out for everyone.
A log flips the shape. The order service appends one event and returns. Inventory reads it within milliseconds, analytics four seconds later, and a consumer that was down catches up on return. Producers and consumers run at their own pace because the broker holds the data in between.
How Kafka Stores and Delivers Messages
What is a topic? A named stream of messages, such as orders, split into a fixed number of partitions.
What is a partition? A log on disk that grows by appending at the end. Each message gets an offset, its integer position in the log. The producer hashes the message key to pick the partition, so equal keys land together; a null key spreads round-robin.
How does a partition live on disk? As segment files; the active one takes appends and rolls at a size limit (1 GB by default) or an age limit. Writes are sequential appends and reads are sequential scans, the pattern disks serve fastest. Data flows through the page cache, so a consumer near the head reads from memory, and sends use zero-copy (sendfile) from page cache to network card with zero trips through user space.
What is a consumer group? Processes that share a group ID and split a topic's partitions between them. Within a group each partition is read by exactly one consumer at a time. Separate groups each receive every message, and each commits its own offsets. A consumer joining, leaving, or missing a heartbeat triggers a rebalance that reassigns partitions and pauses them briefly.
How long does data stay? Retention is by time (7 days by default), by size, or both, dropping the oldest segments. A compacted topic keeps the latest value per key, a changelog of current state; a null value is a tombstone that removes the key.
Ordering: One Partition, One Key
Kafka keeps order inside a partition and makes zero promises across partitions, so the key decides the ordering. A user adds an item, removes it, and checks out. Keyed by user_id, all three events hash to one partition and are read in sequence by the single consumer that owns it. Keyed by a random UUID, the remove may run before the add. Pick the key as the narrowest entity whose events must stay in sequence; a too-broad key such as tenant_id caps one tenant at one consumer's throughput. On the producer, turn idempotence on and keep max.in.flight.requests.per.connection at 5 or below, so retried batches keep their order.
Delivery Semantics
At-most-once. Commit the offset, then process. A crash in between drops the message; fine for metrics.
At-least-once. Process, then commit. A crash in between replays the message, so the handler runs twice. This is the default to design for.
Exactly-once. The idempotent producer tags each batch with a producer ID and sequence number, so the broker discards a retried batch it already wrote. Transactions write to several partitions and commit consumer offsets atomically; read_committed consumers see either all of a transaction or zero of it. Interviewers probe the side effect outside Kafka: if the handler charges a card, a crash after the charge and before the commit still replays. The answer is an idempotent handler that stores an idempotency key with the effect, so the second run finds the record and returns.
Replication and Durability
Each partition has a replication factor, typically 3. The leader serves reads and writes; followers copy its log. The in-sync replica set (ISR) is the leader plus every follower caught up within a lag threshold.
- -
acks=0- send and move on; a broker crash drops messages. - -
acks=1- the leader wrote it; a leader crash before followers copy drops it. - -
acks=all- every replica in the ISR wrote it.
acks=all alone has a hole: an ISR shrunk to the leader means one copy. Pair it with min.insync.replicas=2. With 3 replicas, a write succeeds while at least 2 hold it, so one broker down causes zero interruption. With two brokers down the ISR falls to 1, the broker rejects writes with NotEnoughReplicas, and reads continue from the surviving leader. A leader crash triggers election of a new leader from the ISR within seconds.
Throughput Arithmetic, Lag and Replay
100,000 msgs/s at 1 KB is 100 MB/s. Per day: 100 MB/s x 86,400 s = 8.64 TB, x 3 replicas = 25.9 TB, x 7 days retention = about 181 TB.
Partitions cap parallelism. At 10,000 msgs/s per consumer, 100k/s needs 10 consumers, so 10 partitions; pick 12, which divides evenly by 2, 3, 4 and 6.
Consumer lag is log end offset minus committed offset. Alert on lag in seconds (lag divided by production rate).
Back-pressure is built in: consumers pull, so a slow one just grows lag. Replay is an offset reset to an earlier offset or timestamp, back to the retention limit.
When a Queue Fits Better
A queue such as SQS or RabbitMQ hands each message to one worker and deletes it on ack. It fits job distribution with per-message retry and a dead-letter queue. Kafka fits event streams: many readers, replay, ordering by key.
Schema registry. A registry stores versioned schemas (Avro, Protobuf) and each message carries a schema ID. Backward compatibility means new consumers read old data, forward means old consumers read new data, full means both; the registry rejects a breaking change at registration.
When to Use in Interviews
Reach for Kafka when a design has several independent consumers of one event, needs replay, or needs a buffer between a bursty writer and a steady reader: order pipelines, activity feeds, change data capture. State the partition key, the partition count with the arithmetic, acks=all with min.insync.replicas=2, and the idempotent consumer. Pick a plain queue for job distribution with per-message retries.
Summary
- -The model - Partitions are logs read by offset; sequential IO and zero-copy make it fast.
- -Consumer groups - One partition per consumer within a group.
- -Ordering - Per partition, set by the key.
- -Durability - RF 3, acks=all, min.insync.replicas=2.
- -Semantics - Idempotent handlers for external side effects.
- -Numbers - 100 MB/s, 12 partitions, lag in seconds.
Interview Q&A
- -Why is Kafka fast on disk? - Sequential IO, page cache, zero-copy.
- -How do you order one user's events? - Key by user_id so they share a partition.
- -A consumer dies mid-batch? - Another consumer resumes from the last committed offset and replays the rest.
- -Does exactly-once mean one handler run? - Within Kafka, yes; external side effects need an idempotency key.
- -How many partitions? - Target rate over per-consumer rate, plus headroom: 12 for 100k/s.
Disclosure: I build Pretense, a local proxy that swaps secrets, PII and code identifiers for stand-in tokens before a prompt leaves your laptop, then swaps the identifiers back in the answer while secrets stay as stand-ins. Pasting a producer config with a SASL password into an AI assistant is the moment it earns its keep.
From the System Design Laws newsletter.
