OOM should not be wrapped

[chibenwa]

Describe the bug

When an OutOfMemory exception is raised by the driver, it gets wrapped in a SdkClientException, preventing application error handling to quick in, and for instance shutdown the application.

Expected Behavior

I would expect errors not to be warpped.

Current Behavior

The following exception is logged by my applicative code:

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Java heap space
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:211)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:207)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:179)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
        at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:165)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
          Caused by: java.lang.OutOfMemoryError: Java heap space

Reproduction Steps

Many big object read in parallel compared with the heap size.

Possible Solution

Do not wrap JVM errors

Additional Information/Context

https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3793

AWS Java SDK version used

2.17.198

JDK version used

openjdk version "11.0.16" 2022-07-19

Operating System and version

Ubuntu 20.04.4 LTS

[yasminetalby]

Hello @chibenwa ,

Thank you very much for your submission. I have brought this up to the team and we have added this issue has been added to our backlog. I will post an update here once this is fixed.

Best,

Yasmine.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[BusyByte]

@yasminetalby @chibenwa @davidh44 Has this been back-ported to the V1 java sdk? I searched and couldn't find. We are seeing this on a relatively new version of com.amazonaws:aws-java-sdk-core:1.12.528 for S3 and SQS. It's causing our http connection pools to get into an invalid state:

j.l.IllegalStateException: Connection pool shut down
 at o.a.h.util.Asserts.check(Asserts.java:34)
 at o.a.h.i.c.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269)

Which we've observed for S3 and SQS clients. We have to manually terminate the instances now when we see this. However if AWS didn't catch the OutOfMemoryError and wrap it in SdkClientException we'd just terminate the ecs instance when the OOM occurred. Which the recommended action for error handling is not to handle the OutOfMemoryError and let it shutdown the JVM. Here's the stack trace we are seeing:

j.l.OutOfMemoryError: Java heap space
Wrapped by: c.a.SdkClientException: Failed to sanitize XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
	at c.a.s.s.m.t.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:239)
	at c.a.s.s.m.t.XmlResponsesSaxParser.parseListObjectsV2Response(XmlResponsesSaxParser.java:355)
	at c.a.s.s.m.t.Unmarshallers$ListObjectsV2Unmarshaller.unmarshall(Unmarshallers.java:136)
	at c.a.s.s.m.t.Unmarshallers$ListObjectsV2Unmarshaller.unmarshall(Unmarshallers.java:125)
	at c.a.s.s.i.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
	at c.a.s.s.i.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)
	at c.a.s.s.i.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)
	at c.a.h.r.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)
	at c.a.h.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794)
	at c.a.h.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)
	at c.a.h.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)
	at c.a.h.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
	at c.a.h.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at c.a.h.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at c.a.h.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at c.a.h.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at c.a.h.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at c.a.h.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at c.a.h.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at c.a.s.s.AmazonS3Client.invoke(AmazonS3Client.java:5520)
	at c.a.s.s.AmazonS3Client.invoke(AmazonS3Client.java:5467)
	at c.a.s.s.AmazonS3Client.listObjectsV2(AmazonS3Client.java:1001)

found another instance of it being wrapped as well

j.l.OutOfMemoryError: Java heap space
Wrapped by: c.a.AmazonClientException: Unable to complete transfer: Java heap space
	at c.a.s.s.t.i.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286)
	at c.a.s.s.t.i.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265)
	at c.a.s.s.t.i.UploadImpl.waitForUploadResult(UploadImpl.java:66)