aiokafka
aiokafka copied to clipboard
feat(fetch): add full Fetch v10 support
feat: add FetchRequest v10 support (Kafka ≥ 2.6, KIP-110/320)
Problem
aiokafkaalways sends FetchRequest v4.
When a topic or broker is configured withcompression.type=zstd(default since Kafka 2.6), the broker must deliver Zstd-compressed batches.
Because v4 does not advertise Zstd capability, the broker returnserror_code 76 (UNSUPPORTED_COMPRESSION_TYPE)and the consumer stalls with
Unexpected error while fetching data: UnknownError.
What this PR adds
| Module | Change |
|---|---|
| Protocol | • Implement FetchRequest_v10 / FetchResponse_v10• Set rack_id capability bit → declares Zstd support• Add per-partition current_leader_epoch & log_start_offset• Introduce top-level responses array |
| Fetcher | • Auto-select v10 when broker ApiVersion ≥ 2.1.0 (Kafka 2.6+)• Inject leader_epoch (-1 fallback) per partition• Parse v10 responses and decode Zstd batches • Treat error codes 74 / 75 (FENCED / UNKNOWN_LEADER_EPOCH) as retriable → triggers metadata refresh |
| Tests | • test_fetch_v10_format covers request/response round-trip and happy-path fetch with Zstd |
Result
aiokafka consumers now fetch successfully from Kafka 2.6+ clusters where Zstandard is enabled at broker or topic level-no more silent hangs.
For older brokers (ApiVersion < 2.1.0) the client keeps using v4, so behaviour remains backward-compatible.
Checklist
- [x] Implementation & lint
- [x] Unit tests
- [ ] Docs update (will add after review)
- [x] News fragment
fetch_v10.feature