ruby-kafka

ruby-kafka copied to clipboard

I'm surprised this hasn't come up before because I'm not sure if this ever worked. It appears we need to use v10+ of the fetch protocol to get zstd support (https://github.com/edenhill/librdkafka/pull/2183). Obviously, we can't just bump the version because that would break support for Kafka < 2.1.0. I don't see any examples where we're conditionally using different protocol versions, so I'd like to start a discussion before I start putting together a PR.

• Version of Ruby: 2.7.1
• Version of Kafka: 2.1.0+
• Version of ruby-kafka: 1.3.0

Steps to reproduce

kafka = Kafka.new(...)

# assuming single partition topic foo with zstd compression exists
kafka.deliver_message("bar", topic: "foo")
kafka.fetch_messages(topic: "foo", partition: 0)

Expected outcome

An array of messages is returned

Actual outcome

Unknown error with code 76

Only having compatibility with Kafka 2.1+ would be a non starter at this point, I believe. I could be persuaded otherwise if there's some kind of index of Kafka version deployment?

In general I think it makes the most sense for people to start migrating to rdkafka-ruby or higher level frameworks built on top of it like Racecar in order to get support for more recent features. Ruby-kafka will necessarily lag behind due to not having multi-version support for the Kafka APIs.

We could have some sort of mapping between Kafka versions and various API protocol versions / request / response classes that the client would reference when making requests? But that seems like a pretty heavy refactor.

~~We'd love to start using rdkafka-ruby, but pieces of our logging pipeline use fluent-plugin-kafka which depends on ruby-kafka.~~

It actually looks like Fluentd might have rdkafka-ruby support at this point, so that seems like the way to go and this issue can probably be closed.

Issue has been marked as stale due to a lack of activity.