How to get current offsets of consumer group without subscribing

[yoneal]

First of all thanks for this wonderful project!

In #1161 support for getting the end_offsets was added to get the high watermark of partitions without actually consuming. Is there a way to get the current offset of a consumer group, similar to the CURRENT-OFFSET column of the ff command:

./bin/kafka-consumer-groups.sh --bootstrap-server bootstrap:9092 --describe --group console-consumer-92539
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my_topic                      0          845771          845787          16         consumer-1-4acb3a97-d72e-4c68-805d-c9a8089b5a26   /10.128.0.17                   consumer-1

I seem to be missing it from the documentation..

[jeffwidman]

currently to do this you'll need to directly use KafkaClient and pass the protocol calls directly. This is much easier if you use a broker >= 0.10.2 because then you don't have to identify the group/partition mapping, just use the functionality of KIP-88.

I'm working on a pull request to add this to the Datadog kafka consumer offsets check, so whenever I push that up you'll see how to do it.

Long term, this would go in the admin client, but no one has had the time to write that yet: #935

[cjdudleybw]

Has this moved forwards at all? Been trying to calculate lags using this API, and all I'm missing is the current offset, as of version 1.4.4.
Are there any other ways of establishing lag? Thanks!

EDIT:

Answered my own question; use a KafkaAdminClient, and call list_consumer_group_offsets(). Compare the results to results of KafkaConsumer.end_offsets()

[jeffwidman]

@cjdudleybw see also #1673 (comment) including the linked source code for datadog for fetching the highwater offsets.

If you are monitoring / alerting on this, then you want to directly query the cluster for the high water offsets... IMO too dangerous to trust the KafkaConsumer.end_offsets() because one day there could be a bug in the consumer that prevents it from having an accurate view of the cluster (if the consumer is somehow deadlocked for example) and your alerts may not fire when you most need them.

[piperck]

@jeffwidman
Hi jeff since #1673 fetching the highwater offsets solution can not direct run in 1.4.7?

KafkaAdminClient._send_request_to_node 

has changed in version 1.4.7......

[jeffwidman]

Please open a new ticket, this one is nearly a year old.

Also, there's a working example here of how to check offsets: https://github.com/DataDog/integrations-core/blob/master/kafka_consumer/datadog_checks/kafka_consumer/kafka_consumer.py