elsa icon indicating copy to clipboard operation
elsa copied to clipboard

Advice on how to configure several producers for the same topic

Open fxn opened this issue 3 years ago • 0 comments

I am working on an applications that produces messages to a single topic that has 100 partitions. The workload is high, I think we would have better performance if there were a few more sockets.

Would it be possible somehow to have 10 sockets? I thought about launching 10 supervisors with respective connection names, and then in Elsa.produce select the connection according to the partition. Each connection would be responsible for 10 partitions (disjoint). That way we guarantee order. So, while all supervisors would spawn 100 processes, effectively 90% of them would be idle (this seems a bit weird, but it is a practical consequence).

Makes sense? Do you have a different recommendation?

fxn avatar Nov 03 '20 21:11 fxn