aiokafka icon indicating copy to clipboard operation
aiokafka copied to clipboard

feat(fetch): add full Fetch v10 support

Open mesteruh opened this issue 6 months ago • 0 comments

feat: add FetchRequest v10 support (Kafka ≥ 2.6, KIP-110/320)

Problem
aiokafka always sends FetchRequest v4.
When a topic or broker is configured with compression.type=zstd (default since Kafka 2.6), the broker must deliver Zstd-compressed batches.
Because v4 does not advertise Zstd capability, the broker returns error_code 76 (UNSUPPORTED_COMPRESSION_TYPE) and the consumer stalls with
Unexpected error while fetching data: UnknownError.

What this PR adds

Module Change
Protocol • Implement FetchRequest_v10 / FetchResponse_v10
• Set rack_id capability bit → declares Zstd support
• Add per-partition current_leader_epoch & log_start_offset
• Introduce top-level responses array
Fetcher • Auto-select v10 when broker ApiVersion ≥ 2.1.0 (Kafka 2.6+)
• Inject leader_epoch (-1 fallback) per partition
• Parse v10 responses and decode Zstd batches
• Treat error codes 74 / 75 (FENCED / UNKNOWN_LEADER_EPOCH) as retriable → triggers metadata refresh
Tests test_fetch_v10_format covers request/response round-trip and happy-path fetch with Zstd

Result

aiokafka consumers now fetch successfully from Kafka 2.6+ clusters where Zstandard is enabled at broker or topic level-no more silent hangs.
For older brokers (ApiVersion < 2.1.0) the client keeps using v4, so behaviour remains backward-compatible.


Checklist

  • [x] Implementation & lint
  • [x] Unit tests
  • [ ] Docs update (will add after review)
  • [x] News fragment fetch_v10.feature

mesteruh avatar May 28 '25 11:05 mesteruh