How Kafka maintains the offset across multiple consumer groups ?

Our scenario

Topic T1 is having Partition P1 and P2
consumer group CG1 and CG2 has 1 consumer each
consumer from consumer group 1 is - CG1C1
and consumer from consumer group 1 is - CG1C2 correspondingly

both consumers are subscribed to same topic T1.

Now question is how Kafka maintains the offset across multiple consumer groups. If 1 consumer group is down for few minutes and agian it is backed up then how it will get the message from the last committed read. CG1 is in offset 15 and CG2 before going down committed till offset 5. so when CG2 is up how it will start getting from offset 6 and CG1 is not impacted.

Tagged:

Comments

  • Basically Consumer groups are independent entities. If consumers in the different consumer groups, subscribe to the same kafka topic, there won't be any clash between offset management. Each consumer group will maintain it's own offset in __consumer_offsets topics(this is the compacted internal topic managing the consumer offsets)

    Kafka provides the option to store all the offsets for a given consumer group in a designated broker (for that group) called the offset manager. Whenever a consumer reads a message from a partition, it sends the offset commit request to offset manager. Once the commit request is accepted, the consumer can read the next offset.

    In case of offset commit failure, consumer retries to commit. Each offset commit is maintained per partition.

Sign In or Register to comment.