spring-cloud-aws icon indicating copy to clipboard operation
spring-cloud-aws copied to clipboard

[SQS] Priority for queues while listening

Open aniketbhatnagar opened this issue 4 years ago • 0 comments

Is your feature request related to a problem? Please describe. I would like to define a priority in @SqsListener so that messages from certain high priority queues are given precedence over normal queues.

Describe the solution you'd like The most basic implementation could change SimpleMessageListenerContainer to accept 2 taskExecutors - one for high priority and another for normal priority. SimpleMessageListenerContainer can then use appropriate taskExecutor based on priority defined in @SqsListener annotation.

A more advanced implementation would be like:

  1. Maintain bounded queues for each priority
  2. All AsynchronousMessageListener threads push messages to appropriate bounded queues.
  3. A separate thread polls messages from bounded queues based on priority and submits task to taskExecutor for handling messages

Or we could use streams:

  1. Build a stream for each queue
  2. Combine streams based on priority
  3. Register callback on combined stream that submits task to taskExecutor for handling messages

Describe alternatives you've considered Can't think of any other alternatives

Additional context If we do advanced implementation or use streams, we also fix the current issue with SimpleMessageListenerContainer that polls messages and waits for all of them to be handled before polling for new messages. That delays processing of a queue is a single message takes too long to process. See https://github.com/Mercateo/sqs-utils

aniketbhatnagar avatar Sep 02 '20 17:09 aniketbhatnagar