Error while downloding file using presignedURL + TransferManager (Unable to execute HTTP request: peer not authenticated)

[GRajaMca1193]

Describe the bug

Error while downloading the file from S3 Bucket using PresignedURL + TransferManager, I was able to download the file using PresignedURL to 99% but while downloading for the remaining 1% it's throwing an exception (Unable to execute HTTP request: peer not authenticated). I have tried on various machines with various internet speeds still exactly at the same size it's failing.

Expected Behavior

Using a PresignedURL, we should be able to download the files.

Current Behavior

After 99% download, it's throwing an exception status peer not authenticated.

Steps to Reproduce

AmazonS3 amazonS3 = AmazonS3Client.builder() .withCredentials(new ProfileCredentialsProvider()) .withRegion(Regions.DEFAULT_REGION) .build(); File targetDestinationFile = new File("sample.zip"); PresignedUrlDownloadRequest presignedUrlDownloadRequest = new PresignedUrlDownloadRequest( "presignedURL"); TransferManager transferManager = TransferManagerBuilder.standard() .withS3Client(amazonS3) .withShutDownThreadPools(true) .withExecutorFactory(() -> Executors.newFixedThreadPool(20)) .build(); try { transferManager.download(presignedUrlDownloadRequest, file).waitForCompletion(); } catch (InterruptedException e) { log.error("Error during file download : {}", e.getMessage()); Thread.currentThread().interrupt(); } catch (AmazonClientException e) { log.error("Error during file download : {}", e.getMessage()); }

Possible Solution

Context

Your Environment

• AWS Java SDK version used: 1.12.57
• JDK version used: Java 11 & Java 8
• Operating System and version: Windows 10

[GRajaMca]

I have tried on Linux as well. still the same issues. can someone suggest the best approach to fix this issue?

[debora-ito]

After 99% download, it's throwing an exception status peer not authenticated.

Can you provide the stacktrace?

[GRajaMca]

Hi @debora-ito, Please find the attached stack trace logs logs.txt Also please find the below code which am using it.

`import com.amazonaws.auth.profile.ProfileCredentialsProvider; import com.amazonaws.regions.Regions; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3Client; import com.amazonaws.services.s3.model.PresignedUrlDownloadRequest; import com.amazonaws.services.s3.transfer.TransferManager; import com.amazonaws.services.s3.transfer.TransferManagerBuilder;

import java.io.File;
import java.net.URL;
import java.util.concurrent.Executors;

public class S3PresignedURLDownload {

public static void main(String[] args) {
    AmazonS3 amazonS3 = AmazonS3Client
            .builder()
            .withCredentials(new ProfileCredentialsProvider())
            .withRegion(Regions.DEFAULT_REGION)
            .build();
    TransferManager transferManager = null;
    try {
        File targetDestinationFile = new File("sample.zip");
        String presignedURL = "#ADD_PRESIGNED_URL";
        PresignedUrlDownloadRequest presignedUrlDownloadRequest =
                new PresignedUrlDownloadRequest(new URL(presignedURL));
        transferManager = TransferManagerBuilder
                .standard()
                .withS3Client(amazonS3)
                .withShutDownThreadPools(true)
                .withExecutorFactory(() -> Executors
                        .newFixedThreadPool(20))
                .build();
        transferManager
                .download(presignedUrlDownloadRequest,
                        targetDestinationFile)
                .waitForCompletion();
        shutdownTransferManager(transferManager);
    } catch (Exception e) {
        shutdownTransferManager(transferManager);
        e.printStackTrace();
    }
}

private static void shutdownTransferManager(TransferManager transferManager) {
    if (transferManager != null) {
        transferManager.shutdownNow();
    }
}

}`

[GRajaMca]

Do we have any workaround to fix this issue ?

[debora-ito]

@GRajaMca thank you for providing the stacktrace. There's nothing in your code sample that would indicate a problem, I wasn't able to reproduce the error locally.

So this might be an issue in the network connectivity. Are you using a Proxy? How big is the file you're downloading? Have you tried to increase the limit of the thread pool to more than 20 threads, or not providing a custom ExecutorFactory? There's a line in the stacktrace that shows that 20 connections are being used before the error:

Connection released: [id: 546][route: {s}->https://abc.s3.us-west-2.amazonaws.com:443][total kept alive: 1; route allocated: 20 of 50; total allocated: 20 of 50]

[GRajaMca]

Hi @debora-ito, please find the below comments.

• Am not using a proxy.
• file size is approximately 3-4GB
• Yes I did try to increase the thread pool by more than 20. still the same issues.
• Yes I have also tried without ExecutorFactory and facing the same issues(By default it creates 10 Threads)

Additional information: After multiple attempts of try, twice the download happened completely. but it's completely unstable.

Please find the below-networking criteria I have tried:

• 100Mbps (Duration taken to download 99% = 3-4 mins)
• 50Mbps (Duration taken to download 99% = 8 mins)
• 35Mbps (Duration taken to download 99% = 10 mins)
• 7-8Mbps (Duration taken to download 99% = 15-20 mins)

Pre signed URL expiration duration is 30mins, After the download failed by the S3 client, using the same Pre signed URL I was able to download the files using the browser.

[GRajaMca]

Hi @debora-ito, Any clues or suggestions to fix this issue?

[debora-ito]

@GRajaMca I'm sorry for losing track of this issue. Are you still experiencing the error?

Let us know if you still want us to investigate it.

[github-actions[bot]]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.