aiokafka
aiokafka copied to clipboard
aio-libs
Update `aiokafka` library to support higher client versions > 3
The current implementation of the aiokafka library within the Confluent platform only supports client versions under 3.0 for Python. There is an error reported in aiokafka/util.py that restricts compatibility.
Additionally, the Confluent platform's version interoperability matrix highlights this limitation.
| Confluent Platform | Apache Kafka® | Release Date | Standard End of Support | Platinum End of Support |
|---|---|---|---|---|
| 7.6.x | 3.6.x | February 9, 2024 | February 9, 2026 | February 9, 2027 |
| 7.5.x | 3.5.x | August 25, 2023 | August 25, 2025 | August 25, 2026 |
| 7.4.x | 3.4.x | May 3, 2023 | May 3, 2025 | May 3, 2026 |
| 7.3.x | 3.3.x | November 4, 2022 | November 4, 2024 | November 4, 2025 |
| 7.2.x | 3.2.x | July 6, 2022 | July 6, 2024 | July 6, 2025 |
| 7.1.x | 3.1.x | April 5, 2022 | April 5, 2024 | April 5, 2025 |
| 7.0.x | 3.0.x | October 27, 2021 | October 27, 2023 | October 27, 2024 |
| 6.2.x | 2.8.x | June 8, 2021 | June 8, 2023 | June 8, 2024 |
| 6.1.x | 2.7.x | February 9, 2021 | February 9, 2023 | February 9, 2024 |
We request extending compatibility to include higher client versions (e.g., 3.0 or above). This would allow users to leverage the latest features and bug fixes offered by newer aiokafka client releases.
Hi @Ben621
My company is using aiokafka against AWS MSK 3.6.0 without issue. I think the code you highlight is just happening when you force the api_version to something different to "auto"
Hey @vmaurin
Can please try to print the api_version
In my case we are using 3.4.x,
But I'm unable to connect to get the api_version
But suddenly that connect, I don't know why !!
After. Investigation that work when I set api_version to 2.8.1
Maybe are you familiar with this issue ?
Also I have checked in the GitHub action they are testing client "0.9.0.1" To "2.8.1"
From the codebase, it sounds parse_kafka_version you pointed out is only use when initializing a producer or a consumer with an api_version. If you let it unset like we do (auto), this method is never called. Then when the client (aiokafka) is connecting to the broker (3.6.0), they will negotiate API version for each individual methods part of the kafka protocol.
I guess then, this parameter is to force a specific version of the protocol, but it should not be used, as Kafka protocol is build for backward and forward compatibility (even today version of the broker probably still support clients in 0.11). So my advice is to let this parameter unset
Hey @vmaurin
can you please check what is you api_version when you set it auto ?
In my case after before they upgrade from kafka 2.7 to kafka 3.5 i was able to connect with auto
but after i got this error
Unable to request metadata from "kafkaserver:9093":
suddenly that work once after waiting 2 minutes for connecting and the api_version was 2.6.0
So, when i set api_version= "2.6.0" that connect directly, otherwise got error, or sometime i'm able to connect without knowing why it's happen !!!
@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0
Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.
There is an error reported in aiokafka/util.py that restricts compatibility.
Could you please provide a complete report with the traceback?
@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0
The method you're referring to looks weird to me. In my opinion, we should rather drop support for brokers <0.10.0 and rely solely on what ApiVersions returns.
Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.
I've been meaning to do this in https://github.com/wbarnha/kafka-python-ng for newer Kafka broker versions. If I get a working solution, I'll open a PR here as well.
Any updates on this? Newer versions of confluent-kafka are being deployed within our company and aiokafka is broken without forcing the version in the config.
