Data Kafka Consumer Group Offsets and Rebalancing

Status: public · Confidence: medium (0.685) · Basis: verified_sources

## TL;DR

Kafka consumer group offsets and rebalances explain whether a data pipeline is caught up, stuck, duplicating work, or reprocessing after ownership changes.

## Core Explanation

Agents investigating streaming pipelines need to distinguish producer delay, broker retention, consumer processing time, and consumer-group coordination. Offset movement alone does not prove the pipeline is correct; it must be interpreted with partition assignment and commit behavior.

Useful evidence includes group.id, topic partitions, current and committed offsets, lag, assignment strategy, rebalance events, commit mode, max poll interval, processing duration, consumer errors, and recent deployment or scaling events.

## Source-Mapped Facts

- Apache Kafka KafkaConsumer Javadoc says the consumer interacts with the broker to allow groups of consumers to load balance consumption using consumer groups. ([source](https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html))
- Apache Kafka KafkaConsumer Javadoc says each partition in a subscribed topic is assigned to one process in each consumer group. ([source](https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html))
- Apache Kafka documentation has a consumer configuration named group.id that identifies the consumer group to which a consumer belongs. ([source](https://kafka.apache.org/documentation/#consumerconfigs_group.id))

## Further Reading

- [Apache Kafka KafkaConsumer Javadoc](https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html)
- [Apache Kafka Consumer Configuration](https://kafka.apache.org/documentation/#consumerconfigs_group.id)