Create AdminClient

[jeffwidman]

This is to track creating an AdminClient to mirror the Java AdminClient mentioned in KIP-4

What are your plans on this front?

Are you waiting til the Java crew releases their AdminClient (as mentioned in KIP-4)? Or ready to start this sooner and then cleanup the implementation to spec once they get to it?

I ask because we could use this relatively soon, right now we have a python script calling subprocess to call shell commands and it's pretty janky. But I suspect the Java AdminClient won't actually be released for a bit yet...

[tvoinarovskyi]

Just to remind, there was a ticket #857 for KIP-4 and Create/Delete API.

[jeffwidman]

Oh, I'd missed that, thanks!

[jeffwidman]

Reopening to track the AdminClient creation per #857 (comment)

As part of this, it'd be nice if there was a simple API for returning topic metadata (without having to consume those topics) such as broker offsets, consumer group offsets, and probably the calculated consumer lag.

[tvoinarovskyi]

What about the actual interface for AdminClient? Should we use something like Scala's AdminClient as a reference?

[jeffwidman]

I haven't looked at the Scala admin client, but there has been a fair bit of chatter recently on the Kafka dev list re: what the new Java AdminClient interface should look like. I think, but haven't confirmed that the Java one and the Scala one are different.

[tvoinarovskyi]

By the way, seems like apache/kafka#2472 was merged into Java Client. It's basically what the ticket is about, right?

[jeffwidman]

Kafka 0.11 is live, including the new Java AdminClient. Now just need to port it to kafka-python

[pvanderlinden]

I was wondering if someone has made a start on this? I started looking at it myself, but I'm not that familiar with the kafka-python code yet, to start it decently from scratch.

For people who are interrested in just creating topics, made a quick and dirty script so far: https://gist.github.com/pvanderlinden/3c9b8fc58e07afe2a205d906ba86d9d0
It is missing a lot of error handling though.

[tvoinarovskyi]

I did not look into it that much, but here's links to Java's implementation and javadocs: https://github.com/apache/kafka/tree/0.11.0.0/clients/src/main/java/org/apache/kafka/clients/admin
https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html

It's quite extensive, they run a background thread that does send/receive with NetworkClient (basically our client_async.py). kafka-python basically copies all implementation details from Java, so all internals should be present already.

[pvanderlinden]

I was planning to work on this when I had more time, but I won't be able to. Unfortunately kafka-python doesn't work until #948 is implemented, and I currently don't have the time to spend on implementing that myself (which by the looks of it is pretty big).

[ashsingh21]

Would it make sense to have a thin abstraction for creating topics using CreateTopicRequest for now? I am implementing something like this to create topics using CreateTopicRequest

[llamahunter]

What's the status of this issue? Progress being made?

[dpkp]

I haven't seen any PRs for it. If you want to take a stab, it would be fantastic!

[ashsingh21]

we wrote an admin_client a while back.

[llamahunter]

See #1540

[llamahunter]

Would like some feedback on pull request above. And, well, for it to be accepted. I didn't make python classes for the return results. Perhaps that would be desirable? Also, there are a bunch of protocol structs seemingly missing in kafka-python, so not all of the admin interface could be implemented.

[jeffwidman]

Closing, as the initial interface is merged in #1540 thanks to @llamahunter's great work