afkak
afkak copied to clipboard
Afkak update for latest broker protocol support in broker version `3.4`
I'm documenting this quickly for internet reference and to gather feedback on the plan to address the items outlined here, which also align with those stated in #45.
Our immediate goal is to support the latest broker protocol, which at the time of writing is 3.x. This is no small task, considering we are currently at 0.9.0 and protocol version 0. In the process, we must also support the various intermediary protocols. With this in mind, I propose several changes to simplify the task and maintain backwards compatibility with the existing internal and public APIs throughout afkak.
- [x] Refactor
afkak.commonstructsfrom named-tuples to a class: As mentioned in #85, the change from a named-tuple to a class is not backwards compatible in the current setup. For instance, adding a newtimestampfield to the message/message set struct would be a breaking change. By refactoring to a class, we can make the timestamp field (and all other fields) optional within the struct, thus avoiding breaking changes. - [x] Migrate from setup.py to pyproject.toml as per PEP-621 (completed in #131)
- [x] Set up formatting and update documentation (completed in #131)
- [ ] Port to pytest for the benefit of
fixtures - [x] Add support for KIP-35 - Retrieving protocol version. Before adding support for new protocol versions, the client must be able to query the broker to determine the appropriate version to use.
Changes from current 0.9.0.1
The following are some notable changes and improvements. This list is not exhaustive and is subject to change. It aims to identify some changes in Kafka that could be incorporated into afkak to enhance its functionality:
- Idempotent Producer kip-98: Introduced in Kafka 0.11.0.0, this significant feature enables producers to write to Kafka idempotently, increasing data consistency by reducing duplicates.
- Record Headers: From Kafka 0.11.0.0 onwards, headers can be added to Kafka messages, allowing for the inclusion of arbitrary metadata for purposes such as routing, timestamps, or message schemas.
- Exactly Once Semantics (EOS): Introduced in 0.11.0.0, EOS improves data consistency and reliability by ensuring each message is delivered exactly once.
- KIP-421: This enhancement to the Kafka ListGroups API includes additional metadata fields that provide information about the state of consumer groups, improving monitoring and management.
- KIP-429: Kafka Consumer Incremental Rebalance Protocol (2.3.0) - This proposal aims to reduce rebalance times and enhance overall consumer performance.
- KIP-117: AdminClient API: This new public API facilitates interaction with Kafka for administrative tasks, offering a higher-level abstraction than existing low-level administrative interfaces.
- KIP-339: Create a new IncrementalAlterConfigs API (2.2.0) - This proposal suggests a more flexible method of updating configurations incrementally.
Wishlist
- [ ] Refactor
afkak.kafkacodec: As discussed in #85, this area requires attention.