Implement librdkafka Admin API

[Steveb-p]

librdkafka allows using Topic API in their 1.0.0 release. It would be a great addition to the library since it would allow taking control of Kafka instance using PHP scripts & applications :)

See https://github.com/edenhill/librdkafka/pull/1766 https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_admin.h

I'm not really familiar with programming of C libraries, so I probably won't be able to help with a PR.

[dirx]

You may take a look into the experimental ffi binding based admin client implementation - i am not 100% sure about all interfaces: https://github.com/dirx/php-ffi-librdkafka/tree/master/src/RdKafka/Admin

You may also take a look into confluents go client implementation (also based on librdkafka) https://godoc.org/github.com/confluentinc/confluent-kafka-go/kafka#AdminClient

[nick-zh]

I will check it out, thx @dirx

[nick-zh]

We need to make a decision regarding how to implement the Admin API. So far the spirit has been to stick close to librdkafka, but in the case of the Admin API, i personally think this makes it really confusing. The two examples below, are code snippets what a topic creation would look like.

This example sticks exactly to the Admin API of librdkafka. Downsides i see are:

• that this could be really confusing why we need a producer / consumer first to instantiate an Admin client
• we would have again another NewTopic that does not create a topic. This is already confusing to many new users for the Producer, so this certainly wouldn't help either.
• the syntax is not really nice to read (at least for me)

$conf = new Conf();
$producer = new Producer($conf);
$adminClient = new AdminClient($producer);

$adminOptions = new AdminOptions($producer);
$adminOptions->setRequestTimeout(1000);

$newTopic = new NewTopic($topicName, $numPartitions, $replicationFactor);
$newTopic->setConfig($name, $value);

$adminClient->createTopics($newTopic, $adminOptions);

The Go Client (also by Edenhill), hides some of the complexity and has in my opinion cleaner interfaces and is easier to read. In this case we would:

• hide the RD Client and internally create a Producer to pass, since it has the least overhead.
• Instead of passing an AdminOptions object to every call we make, set options on the client
• instead option classes named after every call, we would have a general option class TopicSpecification which is better understandable (at least to me)

$conf = new Conf();
//create rk internally (producer for less overhead)
$adminClient = new AdminClient($conf);
$adminClient->setRequestTimeout(1000);

$topicSpecification = new TopicSpecification();
$topicSpecification->setTopicName($name); //etc.
$topicSpecification->setConfig($name, $value);

$adminClient->createTopics($topicSpecification);

[nick-zh]

After some discussion, the approach is clear on how to proceed. Since the scope of php-rdkafka is solely to expose librdkafka functionality / API to PHP, we will stick close to the libs implementation. Abstraction of complexity can still be done in PHP libraries (as e.g. php-kafka-lib), to overcome the the points mentioned above.

[zek]

Any progress on this issue? or workaround to create a new topic with PHP? It's not possible to use 2 libraries at the same time (https://github.com/idealo/php-rdkafka-ffi)