spring-cloud-aws icon indicating copy to clipboard operation
spring-cloud-aws copied to clipboard

Enable SqsTemplate to send more than 10 messages at once

Open tomazfernandes opened this issue 1 year ago • 5 comments

As discussed here, currently SqsTemplate fails if more than 10 messages are provided in a batch with an error from AWS SDK.

We can add support for this use case by partitioning the message list in batches of 10, while also sending the batches in parallel.

This change would likely be around here:

https://github.com/awspring/spring-cloud-aws/blob/562b9334821ea99d7eeac886e60dd59ef73451b3/spring-cloud-aws-sqs/src/main/java/io/awspring/cloud/sqs/operations/SqsTemplate.java#L350-L356

tomazfernandes avatar Feb 09 '24 00:02 tomazfernandes

@tomazfernandes I started to work on it...If it's ok, can you assign this issue to me?

levys17 avatar Mar 06 '24 15:03 levys17

@tomazfernandes I'd be happy to provide a PR for this topic, in case you (@levys17) aren't working on it anymore.

techadri avatar Jun 10 '24 19:06 techadri

@tomazfernandes I'd be happy to provide a PR for this topic, in case you (@levys17) aren't working on it anymore.

I didn't have time to finish that and I've barely worked on it in last weeks....so for me it'll be nice if you do that :D

levys17 avatar Jun 11 '24 17:06 levys17

@tomazfernandes Currently, doSendBatchAsync sends all messages in a single batch request using the sqsAsyncClient, ensuring that FIFO queues maintain the order of messages. If we change to partitioning messages and sending batches in parallel, only messages within each batch will retain order, but the order of the batches may vary. What are your thoughts on this?

techadri avatar Jun 12 '24 19:06 techadri

Oh, that's a great point.

I think the ideal solution would be partitioning batches per message group for fifo queues and then sending each batch in parallel.

A simpler solution would be sending batches in parallel for Standard queues and sequentially for FIFO queues.

WDYT?

tomazfernandes avatar Jun 12 '24 19:06 tomazfernandes