aws-sdk-java-v2 icon indicating copy to clipboard operation
aws-sdk-java-v2 copied to clipboard

Published 20 hours ago • verified publisher icon aws

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient

Open ffried opened this issue 1 year ago • 8 comments

Describe the bug

Using the DefaultS3CrtAsyncClient as produced by S3AsyncClient.crtBuilder().build() fails with a StackOverflowError on Alpine when running a ListObjectsRequest.

Using S3AsyncClient.builder().build() (no CRT) succeeds.

Expected Behavior

The CRT client works just as fine as the normal client (if not better 😉).

Current Behavior

A StackOverflowError is raised.

Reproduction Steps

Reproduction files:

Main.java 
package de.sersoft;

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;

public class Main {
    public static void main(String[] args) {
        var bucketName = args[0];
        try (var s3 = S3AsyncClient.crtBuilder().region(Region.US_EAST_1).build()) {
            System.out.println("Listing objects in bucket " + bucketName);
            var objects = s3.listObjects(builder -> builder.bucket(bucketName)).join().contents();
            System.out.println("Objects in bucket " + bucketName + ":");
            for (var myValue : objects) {
                System.out.println("-" + myValue.key());
            }
        }
    }
}
pom.xml 
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>de.sersoft</groupId>
    <artifactId>AWSAlpine</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.22.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.29.2</version>
        </dependency>
    </dependencies>

    <build>
        <finalName>${project.artifactId}</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.6.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals><goal>single</goal></goals>
                        <configuration>
                            <archive>
                                <manifest><mainClass>de.sersoft.Main</mainClass></manifest>
                            </archive>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>
Dockerfile 
FROM --platform=$BUILDPLATFORM maven:3-eclipse-temurin-21-alpine as buildnode

WORKDIR /aws-alpine

COPY . .

RUN --mount=type=cache,target=/root/.m2/repository mvn clean package -DskipTests


FROM eclipse-temurin:21-jdk-alpine
WORKDIR /aws-alpine

COPY --from=buildnode /aws-alpine/target/AWSAlpine-jar-with-dependencies.jar ./

ENTRYPOINT ["java", "-jar", "/aws-alpine/AWSAlpine-jar-with-dependencies.jar" ]

Then building (with buildx) and running the container, passing a bucket name as command will result in a StackOverflowError.

Stack Trace (from stdout) 
2023-12-21T06:50:27.806548589Z Exception in thread "main" java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null
2023-12-21T06:50:27.806597422Z 	at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
2023-12-21T06:50:27.806692589Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
2023-12-21T06:50:27.806713506Z 	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
2023-12-21T06:50:27.806720256Z 	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
2023-12-21T06:50:27.806724006Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806728672Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806730464Z 	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
2023-12-21T06:50:27.806732089Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806772214Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806776714Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806778464Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806780172Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
2023-12-21T06:50:27.806781922Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
2023-12-21T06:50:27.806783672Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
2023-12-21T06:50:27.806785381Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806787006Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806806672Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806820339Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806826006Z 	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
2023-12-21T06:50:27.806828006Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806829797Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806831547Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806833172Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806834797Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
2023-12-21T06:50:27.806836506Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806838131Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806840422Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.807046089Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.807053172Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
2023-12-21T06:50:27.807056589Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
2023-12-21T06:50:27.807058547Z 	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
2023-12-21T06:50:27.807060256Z 	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
2023-12-21T06:50:27.807061839Z 	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
2023-12-21T06:50:27.807063422Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2023-12-21T06:50:27.807072839Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2023-12-21T06:50:27.807074464Z 	at java.base/java.lang.Thread.run(Thread.java:1583)
2023-12-21T06:50:27.807076047Z Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null
2023-12-21T06:50:27.807077714Z 	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
2023-12-21T06:50:27.807080131Z 	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
2023-12-21T06:50:27.807094964Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
2023-12-21T06:50:27.807097506Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
2023-12-21T06:50:27.807099172Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
2023-12-21T06:50:27.807100839Z 	... 23 more
2023-12-21T06:50:27.807627297Z Caused by: java.lang.StackOverflowError
2023-12-21T06:50:27.807646922Z 	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
2023-12-21T06:50:27.807649256Z 	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
2023-12-21T06:50:27.807650922Z 	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
2023-12-21T06:50:27.807652547Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
2023-12-21T06:50:27.807654131Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
2023-12-21T06:50:27.807655756Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
2023-12-21T06:50:27.807657297Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
2023-12-21T06:50:27.807658881Z 	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
2023-12-21T06:50:27.807660464Z 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
2023-12-21T06:50:27.807662006Z 	at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$new$0(BaseAsyncClientHandler.java:66)
2023-12-21T06:50:27.807663631Z 	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
2023-12-21T06:50:27.807761047Z 	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
2023-12-21T06:50:27.807773339Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.807775422Z 	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
2023-12-21T06:50:27.807777172Z 	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
2023-12-21T06:50:27.807778839Z 	at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
2023-12-21T06:50:27.807780506Z 	at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
2023-12-21T06:50:27.807782089Z 	at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
2023-12-21T06:50:27.807789631Z 	at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
2023-12-21T06:50:27.807791297Z 	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onSuccessfulResponseComplete(S3CrtResponseHandlerAdapter.java:106)
2023-12-21T06:50:27.807792964Z 	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:101)
2023-12-21T06:50:27.807794589Z 	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)

Possible Solution

No response

Additional Information/Context

Using a non-alpine image (removing the -alpine suffixes in the build layer and run layer base image) does not result in the same exception.

This issue was originally reported in the aws-crt-java repository: https://github.com/awslabs/aws-crt-java/issues/737 However, I was asked to report it here instead.

AWS Java SDK version used

2.22.2

JDK version used

21.0.1

Operating System and version

Alpine Linux v3.18

ffried avatar Feb 15 '24 06:02 ffried

So this issue occurred for me as well… we are using some modified version of: amazoncorretto:17 jdk-alpine 3.17

I’m just here to mention we switched the image to amazoncorretto 17 jdk ubuntu 20.04 And it worked 🥹

(But of course you can also use non-crt3 as an alternative…)

tedahn avatar Feb 15 '24 07:02 tedahn

@ffried thank you for reaching out - and bonus points for using collapsible sections in the repro steps. We'll investigate.

debora-ito avatar Feb 19 '24 19:02 debora-ito

I am also encountering this issue.

sdk version: 2.25.7 runtime image: eclipse-temurin:17-jre-alpine

JeHuiPark avatar Mar 14 '24 06:03 JeHuiPark

I am encountering this issue with:

  • software.amazon.awssdk 2.25.8
  • software.amazon.awssdk.crt:aws-crt 0.29.11
  • runtime image eclipse-temurin:21-alpine

gweekleyprovidertrust avatar Mar 14 '24 21:03 gweekleyprovidertrust

This problem also occurs when the official Alpine-based Apache Kafka container is used in Eventbridge tests. A simplified version of the failing execution:

try (var ebClient = EventBridgeAsyncClient.builder()
        .region(US_EAST_1)
        .endpointOverride(ENDPOINT)
        .httpClientBuilder(AwsCrtAsyncHttpClient.builder())
        .overrideConfiguration(clientConfig)
        .credentialsProvider(DefaultCredentialsProvider.create())
        .build()
) {
    var entry = PutEventsRequestEntry.builder()
            .eventBusName(EVENTBUS_ARN)
            .source("kafka-connect.eventbridge-e2e-connector")
            .detailType("kafka-connect-eventbridge-e2e")
            .detail("{\"topic\":\"eventbridge-e2e\",\"partition\":0,\"offset\":0,\"timestamp\":1688464495808,\"timestampType\":\"CreateTime\",\"headers\":[],\"key\":null,\"value\":{\"hello\":\"world\"}}")
            .build();
    var request = PutEventsRequest.builder().entries(entry).build();
    logger.info("{}", request);

    try {
        var response = ebClient.putEvents(request).get(SDK_TIMEOUT, MILLISECONDS);
        logger.info("{}", response);
    } catch (InterruptedException | ExecutionException | TimeoutException e) {
        logger.error("Doh!", e);
    }
}

If does not fail when using Java w/ glibc. See attached log files with -Daws.crt.log.level=Debug enabled:

alpine-temurin21.log ubuntu-temurin21.log

The used versions:

  • software.amazon.awssdk 2.24.10
  • software.amazon.awssdk.crt:aws-crt 0.29.13
  • Eclipse Temurin Container: eclipse-temurin:21-jre/eclipse-temurin:21-jre-alpine

Also, it does not fail if AwsCrtAsyncHttpClient is not used.

agebhar1 avatar Mar 22 '24 17:03 agebhar1

@debora-ito any update on this? we're facing the same issue with alpine and temurin 17 and it's blocking us

UditMishraMoodys avatar Apr 03 '24 21:04 UditMishraMoodys

Got the same issue here as well, running on:

  • software.amazon.awssdk:s3 2.25.23
  • software.amazon.awssdk:aws-crt-client 2.25.23
  • runtime image eclipse-temurin:21-jre-alpine

Same issue running with the S3AsyncClient.builder().build() (no CRT) succeeds, running the CRT client it fails. Giving exactly the same error messages as posted by agebhar1.

samleeflang avatar Apr 05 '24 07:04 samleeflang

Same issue, using:

  • software.amazon.awssdk 2.25.18
  • software.amazon.awssdk.crt:aws-crt 0.29.13
  • runtime image amazoncorretto:11.0.20-alpine

KL0nLutiy avatar Apr 11 '24 13:04 KL0nLutiy

Changing the base image for the spring boot app from eclipse-temurin:17-jdk-alpine to eclipse-temurin:17-jdk-jammy works

software.amazon.awssdk 2.26.9 software.amazon.awssdk.crt:aws-crt 0.29.25

vinujan59 avatar Jul 06 '24 06:07 vinujan59

This should be fixed in https://github.com/awslabs/aws-crt-java/releases/tag/v0.30.3. Please try it out, and let me know if the issue persists.

waahm7 avatar Jul 25 '24 20:07 waahm7

@waahm7 I can confirm this to be fixed using the following versions:

  • software.amazon.aswssdk:bom -> 2.26.22
  • software.amazon.awssdk.crt:aws-crt -> 0.30.3

Thank you!

ffried avatar Jul 26 '24 06:07 ffried

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

github-actions[bot] avatar Jul 31 '24 17:07 github-actions[bot]