aiokafka
aiokafka copied to clipboard
Add round robin partitioner
Changes
This adds a round robin partitioner. When message ordering is not a concern, this can increase consumption throughput by ensuring a more unform temporal spacing between messages in a single partition.
As for production throughput, the modified benchmark shows that it is increased by switching to the RoundRobinPartitioner (partition count was manually set to 4):
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "round-robin" --partition -1
Produced 141488 messages in 1 second(s).
Produced 140304 messages in 1 second(s).
Produced 141488 messages in 1 second(s).
Produced 141488 messages in 1 second(s).
Produced 138528 messages in 1 second(s).
Produced 139712 messages in 1 second(s).
Produced 138528 messages in 1 second(s).
Total produced 1000000 messages in 7.14 second(s). Avg 140092.0 m/s
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "round-robin" --partition -1
Produced 140896 messages in 1 second(s).
Produced 136160 messages in 1 second(s).
Produced 139120 messages in 1 second(s).
Produced 140304 messages in 1 second(s).
Produced 141488 messages in 1 second(s).
Produced 142672 messages in 1 second(s).
Produced 140304 messages in 1 second(s).
Total produced 1000000 messages in 7.14 second(s). Avg 139996.0 m/s
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "default" --partition -1 --key
Produced 96425 messages in 1 second(s).
Produced 96715 messages in 1 second(s).
Produced 96860 messages in 1 second(s).
Produced 96570 messages in 1 second(s).
Produced 96860 messages in 1 second(s).
Produced 96280 messages in 1 second(s).
Produced 95700 messages in 1 second(s).
Produced 96135 messages in 1 second(s).
Produced 95555 messages in 1 second(s).
Produced 92075 messages in 1 second(s).
Total produced 1000000 messages in 10.43 second(s). Avg 95880.0 m/s
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "default" --partition -1 --key
Produced 93235 messages in 1 second(s).
Produced 88450 messages in 1 second(s).
Produced 88885 messages in 1 second(s).
Produced 92365 messages in 1 second(s).
Produced 92655 messages in 1 second(s).
Produced 88015 messages in 1 second(s).
Produced 93090 messages in 1 second(s).
Produced 91060 messages in 1 second(s).
Produced 92945 messages in 1 second(s).
Produced 88885 messages in 1 second(s).
Produced 90415 messages in 1 second(s).
Total produced 1000000 messages in 11.00 second(s). Avg 90889.0 m/s
Checklist
- [x] I think the code is well written
- [x] Unit tests for the changes exist
- [ ] Documentation reflects the changes
- [ ] Add a new news fragment into the
CHANGESfolder- name it
<issue_id>.<type>(e.g.588.bugfix) - if you don't have an
issue_idchange it to the pr id after creating the PR - ensure type is one of the following:
.feature: Signifying a new feature..bugfix: Signifying a bug fix..doc: Signifying a documentation improvement..removal: Signifying a deprecation or removal of public API..misc: A ticket has been closed, but it is not of interest to users.
- Make sure to use full sentences with correct case and punctuation, for example:
Fix issue with non-ascii contents in doctest text files.
- name it