Apache Kafka
Apache Kafka is a distributed event streaming platform built around durable, ordered, append-only log partitions. Producers write records to topic partitions; consumers read them sequentially. The broker retains records for a configurable retention period (or indefinitely with compaction).
Core Concepts
Partitions and Ordering
A Kafka topic is split into partitions — each an independent, ordered, immutable sequence of records. Ordering is guaranteed only within a single partition. A producer chooses the partition via a key hash or round-robin.
Replication and ISR
Each partition has one leader replica and zero or more follower replicas. The set of replicas that are fully caught up with the leader is called the ISR (In-Sync Replicas). A write is considered committed once all ISR members acknowledge it (with acks=all). On leader failure, a new leader is elected from the ISR.
Consumer Groups
Consumers form consumer groups. Kafka assigns each partition to exactly one consumer in the group, enabling parallel consumption with per-partition ordering guarantees. Consumer offsets (= position in the log) are committed to an internal __consumer_offsets topic.
Log Compaction
For topics configured with cleanup.policy=compact, Kafka’s background log cleaner periodically rewrites segments to retain only the latest record per key. This provides a “latest value” semantic but through an eventually-consistent background process — not a real-time index.
Key Characteristics
- Append-only log — records are never modified in place.
- Pull-based consumers — consumers control their read position.
- At-least-once by default — exactly-once requires idempotent producers + transactional writes.
- No native point lookups — reading the current value of a key requires scanning the compacted log or maintaining an external index.
- Column-blind — consumers receive entire records regardless of which fields they need.
See Also
- Apache Fluss — streaming storage that adds KV serving and column pruning on top of the log model
- Introduction to Flink — stream processing engine commonly paired with Kafka