SQS Listener Container in Actuator Health Checks

[pluttrell]

We are using the SQS messaging features in Spring Cloud AWS (version 1.2.1 and 2.0.0 M4).

We had a production outage because the service(s) that support the @SqsListener annotation simply didn't complete configuration and logged the following message. More details can be found here:

Ignoring queue with name 'exampleQueue' as it does not exist.

If we had a custom HealthIndicator (as defined here) that could detect this error/failure state and report the service down, then our infrastructure (Kubernetes) would have restarted the app until it reported that it was healthy, thereby eliminating (or minimizing the duration of) the outage.

Is there any bean that I can use to detect this state, and then inject into a custom HealthIndicator? Or any other way to write a custom HealthIndicator to detect unhealthy services that are supporting the SQS messaging?

Also, perhaps a standard HealthIndicator should be included with the artifacts?

[maciejwalkowiak]

If you mean just to check if a queue exists during runtime, you can create custom health indicator like this:

public class SqsQueueHealthIndicator extends AbstractHealthIndicator {

    private final AmazonSQSAsync amazonSQSAsync;
    private final String queueName;

    public SqsQueueHealthIndicator(AmazonSQSAsync amazonSQSAsync, String queueName) {
        this.amazonSQSAsync = amazonSQSAsync;
        this.queueName = queueName;
    }

    @Override
    protected void doHealthCheck(Health.Builder builder) {
        try {
            amazonSQSAsync.getQueueUrl(queueName);
            builder.up();
        } catch (QueueDoesNotExistException e) {
            builder.down(e);
        }
    }
}

and then use it for each queue you need:

@Bean
SqsQueueHealthIndicator q1HealthIndicator(AmazonSQSAsync amazonSQSAsync) {
    return new SqsQueueHealthIndicator(amazonSQSAsync, "q1");
}

@Bean
SqsQueueHealthIndicator sqsDemoHealthIndicator(AmazonSQSAsync amazonSQSAsync) {
    return new SqsQueueHealthIndicator(amazonSQSAsync, "sqs-demo");
}

Having such a health indicator in the library itself is a good idea.

[pluttrell]

Nope, I'm concerned with the container service actually listening. If someone has @SqsListener on a method, they're expecting that the container is listening for messages and delivering them to the code. If that's not happening, they health check should fail because it means the code isn't running as defined/configured.

[maciejwalkowiak]

Ok, I think I got what you mean. Have you tried adding health indicator like:

class SqsListenerHealthIndicator extends AbstractHealthIndicator {

    private final SimpleMessageListenerContainer container;
    private final String queueName;

    SqsListenerHealthIndicator(SimpleMessageListenerContainer container, String queueName) {
        this.container = container;
        this.queueName = queueName;
    }

    @Override
    protected void doHealthCheck(Health.Builder builder) {
        if (container.isRunning(queueName)) {
            builder.up();
        } else {
            builder.down();
        }
    }
}

and then declaring beans:

@Bean
HealthIndicator sqsListenerHealthIndicator(SimpleMessageListenerContainer container) {
    return new SqsListenerHealthIndicator(container, "my-new-queue");
}

[maciejwalkowiak]

You can take a look at more out-of-the-box solution for this on my branch: https://github.com/maciejwalkowiak/spring-cloud-aws/commit/c230496edfd1191f062db96c6a151d7cfd79c4ef

I am not convinced if use case is so common to merge it into Spring Cloud AWS. What do you think @aemruli ?

[aemruli]

@maciejwalkowiak I think it is definitely worth adding, we would need to push it to the actuator module as I want to keep the messaging module clean. But I see potential also for other modules to benefit from health indicator (RDS as an example).

[maciejwalkowiak]

Ok, I'll prepare a PR.