Data Kafka Log Compaction and Tombstones

Status: public · Confidence: medium (0.685) · Basis: verified_sources

## TL;DR

Kafka log compaction and tombstones tell agents whether a stream can be replayed into current state, why old values remain, and how deletes propagate.

## Core Explanation

Compacted topics behave differently from ordinary time-retained event logs. They are often used for changelog, state, and CDC-like streams where the latest value per key matters. A null payload can be meaningful delete evidence, not just a malformed record.

Agents should inspect cleanup.policy, keys, null values, segment age, compaction lag, delete.retention.ms, consumer replay assumptions, sink delete handling, and schema evolution before concluding that a compacted stream lost data.

## Source-Mapped Facts

- Apache Kafka documentation describes log compaction as retaining at least the last known value for each message key within a topic partition. ([source](https://kafka.apache.org/documentation/#compaction))
- Apache Kafka documentation says log compaction gives finer-grained per-record retention rather than coarser time-based retention. ([source](https://kafka.apache.org/documentation/#compaction))
- Apache Kafka documentation describes delete markers for compacted topics as records that have a key and a null payload. ([source](https://kafka.apache.org/documentation/#design_compactionbasics))

## Further Reading

- [Apache Kafka Log Compaction](https://kafka.apache.org/documentation/#compaction)
- [Apache Kafka Log Compaction Design](https://kafka.apache.org/documentation/#design_compactionbasics)