kafka-go icon indicating copy to clipboard operation
kafka-go copied to clipboard

Published 20 hours ago • verified publisher icon segmentio

Sending many messages

Open salakhovilia opened this issue 4 years ago • 1 comments

Describe the bug When I send a large number of messages via WriteMessages and if at least one of the messages is incorrect, then all subsequent messages in the list are rejected

Kafka Version 2.2.1, branch 0.4

To Reproduce

Expected behavior It is expected that an error message will appear in the logs, but the rest of the messages will be processed

Additional context I also studied the source code of the WriteMessages function, it requires significant improvements, this function contains 3 message processing loops, although you can get by with one, this problem becomes critical when you try to send a large number of messages (for example, 10000)

salakhovilia avatar Dec 02 '20 15:12 salakhovilia

Hi @salakhovilia could you please provide a minimal reproducible example? Also what do you mean by "incorrect messages"?

petedannemann avatar Jun 30 '23 17:06 petedannemann