Ian Cooper
Ian Cooper
Hi @honkuan86 - Right. So it may be worth making sure you have reviewed this: https://brightercommand.gitbook.io/paramore-brighter-documentation/brighter-configuration/kafkaconfiguration#offset-management It is possible at high volume that because we only run one commit thread...
@honkuan86 Any more clarity
@honkuan86 When we scale a stream we do via partitioning, not competing consumers. Each partition has a single thread reading it, usually a single process. Kafka polls, so you get...
PS If you have 500 req/sec, and you had 7 consumers (as above) and we assume 7 partitions therefore, you would need to process each request in 14ms to keep...
@honkuan86 Let me know if the partitions change helped you. @phuaxinyuin If you have an amount of records below the batch size left, then you will find the 'at the...
So I have been running the Brighter Kafka sample, which spits messages out continuously. - The offsets seem to be committed. - The messages "behind" figure will rise up to...
OK. I do note, if I shut down the generator with a partial batch, it remains uncommitted prior to exit. I'll see what the defaults for the sweeper are as...
@honkuan86 @phuaxinyuin OK, I think we have a repro and a fix incoming. It is a threading issue with the usage of a Monitor, that is being exited on a...
Hi @honkuan86 and @phuaxinyuin 9.1.20 has the fix for the bug that you reported. If one of you could test it and give me any feedback necessary, so we can...