aiokafka icon indicating copy to clipboard operation
aiokafka copied to clipboard

Add round robin partitioner

Open Arseniy-Popov opened this issue 10 months ago • 0 comments

Changes

This adds a round robin partitioner. When message ordering is not a concern, this can increase consumption throughput by ensuring a more unform temporal spacing between messages in a single partition.

As for production throughput, the modified benchmark shows that it is increased by switching to the RoundRobinPartitioner (partition count was manually set to 4):

$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "round-robin" --partition -1
Produced 141488 messages in 1 second(s).
Produced 140304 messages in 1 second(s).
Produced 141488 messages in 1 second(s).
Produced 141488 messages in 1 second(s).
Produced 138528 messages in 1 second(s).
Produced 139712 messages in 1 second(s).
Produced 138528 messages in 1 second(s).
Total produced 1000000 messages in 7.14 second(s). Avg 140092.0 m/s
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "round-robin" --partition -1
Produced 140896 messages in 1 second(s).
Produced 136160 messages in 1 second(s).
Produced 139120 messages in 1 second(s).
Produced 140304 messages in 1 second(s).
Produced 141488 messages in 1 second(s).
Produced 142672 messages in 1 second(s).
Produced 140304 messages in 1 second(s).
Total produced 1000000 messages in 7.14 second(s). Avg 139996.0 m/s
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "default" --partition -1 --key
Produced 96425 messages in 1 second(s).
Produced 96715 messages in 1 second(s).
Produced 96860 messages in 1 second(s).
Produced 96570 messages in 1 second(s).
Produced 96860 messages in 1 second(s).
Produced 96280 messages in 1 second(s).
Produced 95700 messages in 1 second(s).
Produced 96135 messages in 1 second(s).
Produced 95555 messages in 1 second(s).
Produced 92075 messages in 1 second(s).
Total produced 1000000 messages in 10.43 second(s). Avg 95880.0 m/s
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "default" --partition -1 --key
Produced 93235 messages in 1 second(s).
Produced 88450 messages in 1 second(s).
Produced 88885 messages in 1 second(s).
Produced 92365 messages in 1 second(s).
Produced 92655 messages in 1 second(s).
Produced 88015 messages in 1 second(s).
Produced 93090 messages in 1 second(s).
Produced 91060 messages in 1 second(s).
Produced 92945 messages in 1 second(s).
Produced 88885 messages in 1 second(s).
Produced 90415 messages in 1 second(s).
Total produced 1000000 messages in 11.00 second(s). Avg 90889.0 m/s

Checklist

  • [x] I think the code is well written
  • [x] Unit tests for the changes exist
  • [ ] Documentation reflects the changes
  • [ ] Add a new news fragment into the CHANGES folder
    • name it <issue_id>.<type> (e.g. 588.bugfix)
    • if you don't have an issue_id change it to the pr id after creating the PR
    • ensure type is one of the following:
      • .feature: Signifying a new feature.
      • .bugfix: Signifying a bug fix.
      • .doc: Signifying a documentation improvement.
      • .removal: Signifying a deprecation or removal of public API.
      • .misc: A ticket has been closed, but it is not of interest to users.
    • Make sure to use full sentences with correct case and punctuation, for example: Fix issue with non-ascii contents in doctest text files.

Arseniy-Popov avatar Jan 27 '25 21:01 Arseniy-Popov