KIP-62 / KAFKA-3888: Allow consumer to send heartbeats from a background thread

[jeffwidman]

Allows consumers to take their time processing messages without being timed out from their consumer group.

max_poll_records is a decent workaround for most of the pain here, but it'd still be nice to add this for consumers that have inconsistent message processing times.

Related issues: #872, #544

[tvoinarovskyi]

Hey @jeffwidman do you plan to do any work on this? If not I probably can take it on the weekend.

[jeffwidman]

No, I'm afraid I won't have time to work on this anytime soon... if you have time, go for it!

Only comment--and you probably already know this--is as far as implementation, the closer it mirrors the Java consumer (without the bugs 😉 ), the better IMHO. Among other things, it makes it so config tuning advice from the Kafka mailing list etc is easier to translate to kafka-python.

[jeffwidman]

We were just discussing this internally at my day job... since Python has the GIL, how would this background thread ever issue a heartbeat if the main consumer thread is CPU-bound in it's message processing?

Ie, the main thread would occupy the CPU the whole time, and the background thread wouldn't get scheduled in to issue the heartbeat unless the message processing was I/O bound.

[tvoinarovskyi]

Hey @jeffwidman
Sorry, but GIL will not block you from doing IO in a separate thread. At least if you are not doing something crazy like passing all messages in a pure C code, that processes them. Python will yield GIL after some time (python3) or instruction count (python2), so as long as you are not stuck in C code it should be OK.

[jianbin-wei]

I agree that in most cases the GIL would not prevent the heart-beat background thread from doing its job.

[jeffwidman]

@tvoinarovskyi are you still planning on working on this?

It's not something I can tackle in the near term, just wanted to check in to see where you're at on it as it would be a very useful addition...

[tvoinarovskyi]

@jeffwidman Hey, sorry, but not seems possible in near term. I did look at how Java client does it, quite a hassle with locking, not sure I can do it properly in my limited free time.

[oscarbatori]

@jeffwidman @Drizzt1991 - is there a pull request for this? We would quite like to leverage KIP-62 functionality in our organization.

[tvoinarovskyi]

@oscarbatori Sorry, nothing too good to show...

[pvanderlinden]

Is there any work around for this problem? I just switched to kafka-python from a different client, only to discover this problem. The problem I have that one of the consumers does a lot of heavy processing taking anything from 10 second to 5 minutes, this means we have to set the session time out high, meaning recovering from crashed consumers will take forever.