beats icon indicating copy to clipboard operation
beats copied to clipboard
verified publisher icon elastic

Update Kafka dependencies to prepare for KIP-896

Open nhaq-confluent opened this issue 1 year ago • 3 comments

Hello!

I am from Confluent, an organization contributing to Apache Kafka, and wanted to raise a request in regards to KIP-896.

In short, KIP-896 removes compatibility of certain client protocol API versions in Apache Kafka(AK) 4.0, being targeted for around January 2025. Since there is a dependency on Kafka in beats (in particular to a forked version of the Sarama client) I wanted to raise awareness that there could be some changes necessary to ensure a smooth experience for those using beats and Kafka. Here are the recommendations.

  1. The latest versions of beats (7.X and higher) appear to be using sarama 1.29.1, which is good enough from the version perspective, however the Kafka version would need to be set to 2.1.0 manually by users to make this work. Recommend updating the default to be 2.1.0 or higher.

  2. Recommend updating the sarama dependency to the latest version possible in future beats versions

  3. Updating the beats docs, advising users of this situation and to set the Kafka version config to 2.1.0

Thank you!

nhaq-confluent avatar Oct 07 '24 22:10 nhaq-confluent

Reference to the PR where the default for Sarama was changed: https://github.com/IBM/sarama/pull/2574

ijuma avatar Oct 07 '24 22:10 ijuma

@ruflin @andrewkroh could you please comment on this issue?

nhaq-confluent avatar Oct 22 '24 14:10 nhaq-confluent

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

elasticmachine avatar Oct 24 '24 04:10 elasticmachine

Hello @nhaq-confluent Thanks for raising this issue and bringing some context to it. We are currently discussing about what would be the best timing to change the default version to 2.1.0 as suggested.

pierrehilbert avatar Oct 24 '24 05:10 pierrehilbert

Thanks. One important detail is that 2.1.0 is truly ancient at this point, it was released 6 years ago (in Nov 2018). So, we should consider changing the default for older releases too to avoid the situation where they are completely broken with Apache Kafka 4.0 (due in 2-3 months).

ijuma avatar Oct 31 '24 14:10 ijuma

Hello @nhaq-confluent @ijuma, is it possible to have access to 4.0.0 SNAPSHOT version for testing? This would help us perform upgrades and test it locally in our labs. TIA

khushijain21 avatar Nov 12 '24 09:11 khushijain21

Closing this as completed by

upgrades sarama version on elastic fork - https://github.com/elastic/sarama/pull/19 Update beats to latest sarama version - https://github.com/elastic/beats/pull/41655 set default kafka protocol version 2.1.0 - https://github.com/elastic/beats/pull/41662 upgrades the documentation - https://github.com/elastic/beats/pull/41540

khushijain21 avatar Dec 04 '24 05:12 khushijain21

Hello @nhaq-confluent @ijuma, is it possible to have access to 4.0.0 SNAPSHOT version for testing? This would help us perform upgrades and test it locally in our labs. TIA

This has now been merged to the 4.0 branch of Apache Kafka: https://github.com/apache/kafka/pull/18291

ijuma avatar Dec 21 '24 08:12 ijuma