MUST READ Architecture
Kafka: Partitioned, Replicated Commit Log Architecture
Kafka is a partitioned, replicated commit log. Producers append, consumers read by offset at their own pace, and retention makes slow readers and replays free for the writer. Ordering holds per partition, durability comes from acks=all over three replicas; the design choices are the partition key, the partition count, and duplicate handling.