Kafka 4.X.X Kraft and scram auth failed

[nicolas-fidel-wmx]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Impossible to connect to a Kafka 4.x.x cluster and the latest release of vector with a TLS kraft cluster and SCRAM authentication. There's a problem in rust rdkafka.

The rdkafka crate fix was introduced in https://github.com/fede1024/rust-rdkafka/commit/3a5ad932670c60875cd23aeca0716ffaade236f2, which updates librdkafka from 2.4.0 to 2.10.0. Using rdkafka crate with the dynamic-linking feature and librdkafka 2.8.0 (current version in Debian stable) also works fine.

Configuration

Version

v0.48

Debug Output

Example Data

No response

Additional Context

No response

References

No response

[pront]

Hi @nicolas-fidel-wmx, thank you for the report. Would you be interested in updating the kafka dependency in Vector?

[nicolas-fidel-wmx]

I'll let you do the PR

[viktordebulat]

rdkafka bump to 2.10.0 brakes auth for <4.x.x Kafka Currently, I keep vector fork with bumped rdkafka for envs with 4.0.x Kafka and official image for <=3.9.x

[dratasich]

Same problem here. Since 4.0.0 update our latest vector debian image vector:0.49.0-debian gets auth errors.

In this kcat issue #462 it says its from librdkafka 2.6.1 onwards. We can reproduce errors with kcat on Ubuntu LTS which has librdkafka 2.3.0 but is working on Arch with librdkafka 2.11.1

~~An update to a newer debian image would help. At least trixie has librdkafka 2.8.0 this works for us (tried with golang 1.24.6-trixie)~~ ...missed that rdkafka-sys comes with bundled librdkafka...

[LordGaav]

Manually updating rdkafka to 0.38.0 and rdkafka-sys to 4.9.0-2.10.0 fixes the issue for me.

[Ra7uR]

Manually updating rdkafka to 0.38.0 and rdkafka-sys to 4.9.0-2.10.0 fixes the issue for me.

Could you please advise how did you do that? Like using this repository or just by replacing cargo.toml file?

[LordGaav]

Manually updating rdkafka to 0.38.0 and rdkafka-sys to 4.9.0-2.10.0 fixes the issue for me.

Could you please advise how did you do that? Like using this repository or just by replacing cargo.toml file?

I just changed the dependency in Cargo.toml and called cargo build.

diff --git a/Cargo.toml b/Cargo.toml
index e05a78d8d..4767ff43a 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -400,7 +400,7 @@ pulsar = { version = "6.3.1", default-features = false, features = ["tokio-runti
 quick-junit = { version = "0.5.1" }
 rand.workspace = true
 rand_distr.workspace = true
-rdkafka = { version = "0.37.0", default-features = false, features = ["curl-static", "tokio", "libz", "ssl", "zstd"], optional = true }
+rdkafka = { version = "0.38.0", default-features = false, features = ["curl-static", "tokio", "libz", "ssl", "zstd"], optional = true }
 redis = { version = "0.32.4", default-features = false, features = ["connection-manager", "sentinel", "tokio-comp", "tokio-native-tls-comp"], optional = true }
 regex.workspace = true
 roaring = { version = "0.11.2", default-features = false, features = ["std"], optional = true }

[Ra7uR]

@LordGaav Thank you a lot