java.lang.IllegalStateException: Connection pool shut down

[ragnar-lothbrok]

I am using amazon-sqs-java-messaging-lib-1.0.8.jar. When I am sending a payload to SQS I am getting java.lang.IllegalStateException: Connection pool shut down exception.

Describe the bug When sending payload to SQS Queue getting below exception intermittently java.lang.IllegalStateException: Connection pool shut down org.apache.http.util.Asserts.check(Asserts.java:34) org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191) …n.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:267) sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) …ttp.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) com.amazonaws.http.conn.$Proxy191.requestConnection(Unknown Source) org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) …g.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) …rg.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) …rg.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) …mazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) …ttp.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297) …ws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) …zonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) …http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) …mazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) …onaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) …p.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2215) com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2182) com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2171) …zonaws.services.sqs.AmazonSQSClient.executeGetQueueUrl(AmazonSQSClient.java:1163) com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:1136) …amessaging.AmazonSQSMessagingClientWrapper.getQueueUrl(AmazonSQSMessagingClientWrapper.java:294) …amessaging.AmazonSQSMessagingClientWrapper.getQueueUrl(AmazonSQSMessagingClientWrapper.java:265) com.amazon.sqs.javamessaging.SQSSession.createQueue(SQSSession.java:636) …rt.destination.DynamicDestinationResolver.resolveQueue(DynamicDestinationResolver.java:85) …tion.DynamicDestinationResolver.resolveDestinationName(DynamicDestinationResolver.java:59) …tination.JmsDestinationAccessor.resolveDestinationName(JmsDestinationAccessor.java:115) org.springframework.jms.core.JmsTemplate.lambda$send$3(JmsTemplate.java:585) org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:504) org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:584)

Expected Behavior Ideally, I should get 200 and should be able to send the payload successfully to SQS.

Current Behavior Getting java.lang.IllegalStateException: Connection pool shut down Exception.

Your Environment amazon-sqs-java-messaging-lib-1.0.8.jar aws-java-sdk-sqs-1.11.832.jar Java 11 Linux

public JmsConfigurataion(JobsProperties sqsProperties) {

    EndpointConfiguration endpointConfiguration;
    endpointConfiguration = new EndpointConfiguration(
        endPoint + queueName, region);
    SQSConnectionFactory = connectionFactory = new SQSConnectionFactory(new ProviderConfiguration(),
        AmazonSQSClientBuilder.standard().withCredentials(
            new AWSStaticCredentialsProvider(new BasicAWSCredentials(
                accessKey(),
                secretKey())))
            .withEndpointConfiguration(endpointConfiguration).build());
}

@Override
@Bean(name = "jmsJobsContainerFactory")
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {

    return super.jmsListenerContainerFactory();
}

@Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
    DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
    factory.setConnectionFactory(this.connectionFactory);
    factory.setDestinationResolver(new DynamicDestinationResolver());
    factory.setConcurrency("3-10");
    factory.setMessageConverter(messageConverter());
    factory.setErrorHandler(new ErrorHandler() {
        @Override
        public void handleError(Throwable t) {
            logger.error(t.getMessage(), t);
        }
    });
    return factory;
}

@Bean(name = "jmsJobsTemplate")
public JmsTemplate defaultJmsTemplate() {
    JmsTemplate jmsTemplate = new JmsTemplate(this.connectionFactory);
    jmsTemplate.setMessageConverter(messageConverter());
    return jmsTemplate;

}

Link - https://github.com/aws/aws-sdk-java/issues/2497

[rndbblnn]

following. same issue happened here on our production environment

[ragnar-lothbrok]

following. same issue happened here on our production environment

How did you resolve?

[ragnar-lothbrok]

@rndbblnn In this (https://github.com/aws/aws-sdk-java/issues/2337#issuecomment-807985862) some folks mentioned increasing heap but it didn't work.

[geheim01]

I am facing the same problem. Has anyone found a solution?

[ddrouin]

Same here with 10 long lived threads long polling indefinitely - and then... We would see this on shutting down an instance but now are also seeing this oocur on one of our dev ecs instances after about an hour running 20 sec long polls. CPU spike and kill the app is killed. Not sure if this itself is the cause or a side effect of the app shutting down yet.

[kevinashaw]

We are seeing a similar error with a Java 15 application. We are not using Spring or SQS, but are seeing the error on DynamoDB requests. This is on a production server instance after about 3 days of uptime and about 477K writes to DyanmoDB. We are getting errors on S3 writes also.
We are using aws-java-sdk-core version 1.12.51 and dynamodb version 2.17.24, as defined in our pom.xml file. The DDB error is:

Failed to batchWriteItem to DDB; General Exception received.
java.lang.IllegalStateException: Connection pool shut down
	at org.apache.http.util.Asserts.check(Asserts.java:34)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269)
	at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
	at com.amazonaws.http.conn.$Proxy9.requestConnection(Unknown Source)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1343)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:6214)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:6181)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeBatchWriteItem(AmazonDynamoDBClient.java:768)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:732)
	at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:113)
	at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:53)
	at com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
	at com.algoint.XXX.DDB.writeRecord(DDB.java:155)
	at com.algoint.XXX.timers.Timer_WriteDDB_GeneralStatus$1.run(Timer_WriteDDB_GeneralStatus.java:79)
	at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
	at java.base/java.util.TimerThread.run(Timer.java:506)

[AnkitKadam567]

Does anyone find any solution here? I'm facing similar issue while fetching objects from S3

[lucaalex87]

Any updates on this?

According to some AWS Documentation, if we increase the number of MaxConnections this should resolve itself or at least happen less often.

Another fix is to catch the IllegalStateException, and renew your client at that point and try again.

Can this be built into the library maybe?

Facing this in PROD also, when there is high concurrency. Affects SQS and S3Client (I know this issue is for SQS only).

Thanks for any potential future update :)

BTW this is how I create the client to allow for more connections - yes it's an app wide Bean. I'll also try to create it manually not as a bean, but with a regular singleton. (and maybe have a method that recreates the client, and call that in catch blocks... or something :) like that).

@Bean
public SqsClient sqsClient() {

	SqsClient sqs = SqsClient.builder()
			.region(Region.of(region))
			.credentialsProvider(() -> AwsBasicCredentials.create(awsId, awsKey))
			.httpClient(ApacheHttpClient.builder().maxConnections(150).build())
			.build();

	return sqs;
}

[mrclrchtr]

We have experienced the same problem and would be happy about an update or workaround.

[krishnar700]

I am facing the same problem. while uploading the objects into S3 by using S3Publisher plugin through the Jenkins.

[Silence-Wang]

I sold my problem that be same with you. I shutdown my client manually. Because the whole project use the only one client, when i start my server, it can be get data once, then i try to get the data from the same interface, it will throw this exception.

So I just remove the code "client.shutdown()", It's OK.

[antique3158]

Does anyone find any solution here?

[lucaalex87]

I made this workaround which seems to work for my use case:

I run all my SQS calls through something like this: the new SQSClient method just creates a new client. this is like a wrapper over the AWS class with a retry built in. whatToThrow just converts everything to a RuntimeException.

public SendMessageResponse sendMessage(SendMessageRequest request) {
		return runAWSCall(() -> getSQSClient().sendMessage(request));
	}

private <T> T runAWSCall(Callable<T> toCall) {
		try {
			return toCall.call();
		} catch (IllegalStateException | Error illegalStateException) {
			// new client and retry ONCE only
			sqsClient = newSQSClient();
			try {
				return toCall.call();
			} catch (Exception e) {
				throw whatToThrow(e);
			}
		} catch (Exception e) {
			throw whatToThrow(e);
		}

	}

You can also specify more connections for the SQS connection pool:

		private SqsClient newSQSClient() {

		SqsClient sqs = SqsClient.builder()
				.region(Region.of(region))
				.credentialsProvider(() -> AwsBasicCredentials.create(awsId, awsKey))
				.httpClient(ApacheHttpClient.builder().maxConnections(150).build())
				.build();

		return sqs;
	}

I have a similar wrapper over the S3Client too.

more info here: https://blog.adebski.com/posts/apache-http-client-shutting-down/ I got the idea from the link above: "one of the possible solution could be:

Implement a wrapper around the client libraries that use Apache HTTP under the hood. That wrapper would handle the java.lang.Errors and replace old clients with new clients."

[antique3158]

Hi @lucaalex87 I use the @SqsListener annotation to consume the sqs message, so what should I do with this retry.

[yasamin93]

Thanks @lucaalex87. My issue was with another AWS service, Rekognition. but setting the .httpClient(ApacheHttpClient.builder().maxConnections(150).build()) solved my issue. I also had to override the config.

import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.core.retry.RetryMode;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.http.apache.ApacheHttpClient;

RekognitionClient.builder()
                .region(Region.of(region))
                .httpClient(ApacheHttpClient.builder().maxConnections(150).build())
                .overrideConfiguration(
                        ClientOverrideConfiguration.builder().retryPolicy(RetryMode.STANDARD).build());