Have one consumer for retry topics

[garyrussell]

Discussed in https://github.com/spring-projects/spring-kafka/discussions/2146

^{Originally posted by psh05 March 4, 2022} Currently there is one consumer per topic. If an app consumes from multiple topics which will implement multiple retries and there are multiple instances of this application deployed, then the number of consumers created will be quite large. e.g. If we have 3 topics, each with 3 retries and 1 DLT. Then we will have 3*4= 12 consumers and if we have 50 instances then thats 12 * 50 = 600 consumers.

@tomazfernandes - Is this possible? or is there a way round this?

[garyrussell]

Deleted the previous comment; 2.9 will not be tied to the Boot 2.7 release train.

[tomazfernandes]

One interesting follow-up for this, or maybe an even better solution, would be handling all retries within a single retry topic, relying on partitions for different delay values instead.

A topic with 5 delays for example might use a single retry topic with 5 partitions. By default we could have a single consumer to consume all partitions, then users could increase that up to 5 concurrent consumers. If users want more concurrency, they might double / triple ... the number of partitions so that each delay value gets as many partitions and consumers.

While this should not require any major changes in the feature's architecture, it should take quite a few adjustments in components, so quite a bit of work. But definitely doable, and I think should make it a much simpler and resource-optimal solution for users, since they'd not have to setup so many topics.

[tomazfernandes]

We might even have a single retry topic for all topics within an application 🤯 Though that seems like more complicated for users to get the number of partitions right, so maybe an overkill. Also, would need to keep track of the proper methods to process records, so no-no 😄. A fun thought though.