Remove dependency on sarama-cluster (Kafka driver)

[yurishkuro]

• partitions not picked up on rebalance: https://github.com/bsm/sarama-cluster/issues/235
• partitions are can be held even when no work is being done by client: https://github.com/bsm/sarama-cluster/issues/261
• potential deadlock when rebalancing: https://github.com/bsm/sarama-cluster/issues/255#issuecomment-413967695

[yurishkuro]

Transferring #1090 by @vprithvi here:

Use high level consumer groups from Shopify/sarama#1099 instead of sarama-cluster. This is a rewrite of high level consumer functionality and can side step the issues described in #1052

[marqc]

Instead of sarama You can take a look at https://github.com/confluentinc/confluent-kafka-go It's a light wrapper for C implementation of kafka client (librdkafka) and AFAIK is more stable/reliable than shopify/sarama.

[vprithvi]

@marqc Thanks for the suggestion - we haven't yet explored it because we weren't familiar with how the build and deployment process changes when depending on a native bindings.

[yurishkuro]

My vote is against libraries that depend on native bindings. Not only do they complicate the build in CI, they increase the barrier to entry for new contributors.

[isaachier]

Ironically, given my C++ background, I happen to agree. Also, go get is completely useless for any Go package using non-Go build dependencies.

[ledor473]

👍 on not using the C wrapper.

We used it in a project internally and had a hard time making it work on older kernel (especially having a statically-linked binary)

[clock21am]

is anyone working on this issue, i have bit experience working with Kafka and sarama can take this task if this is okay ?

[abhishek-j16]

Is anyone actively working on this? We still see the issue with Jaeger version 1.56(latest as of today).

We see that rebalancing is not trigerred upon change in partitions, eventually causing data loss.