Unexpected in-flight messages in SQS during restart/shutdown

[merickson-vg]

Type: Bug

Component: SQS

Describe the bug io.awspring.cloud 2.3.1 spring-cloud-aws-messaging, spring-cloud-aws-dependencies, spring-cloud-aws-autoconfigure spring boot 2.4.5

Problem: during app restarts, queued SQS messages go in flight without acknowledgement in the app. During app shutdown, it appears that the polling mechanism is still pulling messages from SQS after the handler has been shut down. So what we're seeing is:

Test is run in the AWS environment A queue with thousands of messages is prepared The @SqsListener method is set up to immediately log the message and then wait 1000ms to simulate latency of real world tasks Queue is configured with wait time of 0 zero seconds (i.e., short polling is active) App shutdown is initiated During shutdown, clusters of messages go in flight without creating a log in the app On completion of restart, the messages remain in flight until the visibility window expires

Restarting the app repeatedly produces more unprocessed messages in flight. Note: the issue is intermittent and appears somewhat dependent on timing of the shutdown command. To reproduce the issue, you may need to restart the app multiple times.

This presents an issue for us during deployments (we do rolling restarts on load balanced instances). We could set the queues to retry on failure, but there are some tasks (sending emails, etc) where automatic retrying is not the desired behavior.

Sample I will add a github link here to a proof of concept app demonstrating the issue soon.

[maciejwalkowiak]

@merickson-vg can you prepare a sample that reproduces this issue and ideally verify if it still happens with 3.0-SNAPSHOT (or M2 once released)?

[maciejwalkowiak]

Close due to lack of feedback