aws-sdk-java-v2 icon indicating copy to clipboard operation
aws-sdk-java-v2 copied to clipboard
verified publisher icon aws

Unable to use CompleteMultipartUploadRequest with presigned URL

Open jeffcfbr opened this issue 2 years ago • 6 comments

Describe the bug

I need to make a Multipart Upload using presigned URLs. I understand I need to create one URL for the CreateMultipartUploadRequest, one for each UploadPartRequest, and one for the CompleteMultipartUploadRequest.

I am able to make the create-mpu and the upload-part, but I always get SignatureDoesNotMatch for the complete-mpu, no matter what I try.

Expected Behavior

I expected that once I create an XML in the correct format, I'd be able to call the presigned URL for the CompleteMultipartUploadRequest and it would successfully create the target file.

Current Behavior

I am able to successfully generate the presigned URL for the complete-mpu. However, when I try to call it, I always get error SignatureDoesNotMatch.

(actual bucket name replaced with mybucket)

curl \
  --request POST \
  --location 'https://mybucket.s3.eu-central-1.amazonaws.com/file?uploadId=pN8ZWQSSRiGpbk0rPZcRsOjyYFJF8JyKzkGiGqqrOEEviA825N8eQHuUiiUIPrSndk_B1PK3CwzBXXdQZOEJucKtSeHhX6i98Xcf1ewBBkFNItLK728Bnw0vinmtI4MyAr3R5pgWQ8k3GFN7KSuJ7XJZ2TVR9wWRHGlM9yhSgrbPZX48vOQooTtgQpXz475d&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEN7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDGV1LWNlbnRyYWwtMSJGMEQCIFVGYSxM%2B%2BYU7gGlgXtIgmA%2Fy5UmnoocvXWw4BxxK6PUAiA71ZaSEgzlB9MoRU4KwSnJnRSRDfkIQQ2CqlCtCpM%2F%2FyqxAghnEAIaDDg4NTQwNjE3ODUyMiIMNI5dFpy5G7KZnVncKo4C7X%2B%2FnHFz3CrJ7BiK01hNr5yQbg1Ppz9Wb18FyhOd65MLmphhMvNAZGNzW18fMcIdM7YnvTF6K1MP%2B6vH38xEQpAehZe%2BXA8mt7%2FEHoA%2Bo%2BsjockIQK%2FhGbLD0zpjqLWEngn8XxWT9c9UjTbX2CHotlelVkgLCuz6iAv1a1kIb9gGnKBUjcAB7HZFwlVghC%2FrnnGqx%2FqrVLGu9B0aPQDT4PkGDQgU1a878SE6D3yRBJ1lqp64ZTMsC1p3ibZk8Qtrvg6N%2FWGkYuYoH%2BAtzAIby%2BctqvhU4HPnELzG0OVmF8K7VOa1oqxSpAvQFd4sqj6bHfcxcd1wvVqz6vUT7sY44Z5n%2F5KMYcwSFI5tVrxqMOfk1qUGOp4B85T13aDycd2vXm9dd7G0lFhJA3MEV8%2Fwxk89KzccV%2Fe6B%2BzeTDI8utqNQwJdQFzwtslD8O38wSA7ZD8sjrhP9ifVXK4TONWcW34ui5s0Aa2SXd4XQlHmoYhFdn8h55ovqpo3FNQOoFUJ9TeQK29tXh5gsmWOKAh3%2BrfbNUXYgxVNzkmmtb%2FBkOjfWRDRyOhe21ebtBej7Mp5il%2FeCCw%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230717T212813Z&X-Amz-SignedHeaders=content-length%3Bcontent-type%3Bhost&X-Amz-Expires=3600&X-Amz-Credential=[REDACTED]%2F20230717%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=407de5295a92f1ce9c12db5fb1acff8a5cdb67d26965651dd0bc5c3c316fcec0' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data '<CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Part><ETag>"dfd1ea4fc17b35d96124927b2c154823"</ETag><PartNumber>1</PartNumber></Part></CompleteMultipartUpload>'

<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
  <AWSAccessKeyId>[REDACTED]</AWSAccessKeyId>
  <StringToSign>AWS4-HMAC-SHA256
20230717T212813Z
20230717/eu-central-1/s3/aws4_request
16fce72535f4d22efdbb3537b9e43aef33c5bf995e710bbd9b76d60a3c5becf0</StringToSign>
  <SignatureProvided>407de5295a92f1ce9c12db5fb1acff8a5cdb67d26965651dd0bc5c3c316fcec0</SignatureProvided>
  <StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 33 30 37 31 37 54 32 31 32 38 31 33 5a 0a 32 30 32 33 30 37 31 37 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 31 36 66 63 65 37 32 35 33 35 66 34 64 32 32 65 66 64 62 62 33 35 33 37 62 39 65 34 33 61 65 66 33 33 63 35 62 66 39 39 35 65 37 31 30 62 62 64 39 62 37 36 64 36 30 61 33 63 35 62 65 63 66 30</StringToSignBytes>
  <CanonicalRequest>POST
/file
%3CCompleteMultipartUpload%20xmlns=%22http%3A%2F%2Fs3.amazonaws.com%2Fdoc%2F2006-03-01%2F%22%3E%3CPart%3E%3CETag%3E%22dfd1ea4fc17b35d96124927b2c154823%22%3C%2FETag%3E%3CPartNumber%3E1%3C%2FPartNumber%3E%3C%2FPart%3E%3C%2FCompleteMultipartUpload%3E&amp;X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=[REDACTED]%2F20230717%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20230717T212813Z&amp;X-Amz-Expires=3600&amp;X-Amz-Security-Token=IQoJb3JpZ2luX2VjEN7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDGV1LWNlbnRyYWwtMSJGMEQCIFVGYSxM%2B%2BYU7gGlgXtIgmA%2Fy5UmnoocvXWw4BxxK6PUAiA71ZaSEgzlB9MoRU4KwSnJnRSRDfkIQQ2CqlCtCpM%2F%2FyqxAghnEAIaDDg4NTQwNjE3ODUyMiIMNI5dFpy5G7KZnVncKo4C7X%2B%2FnHFz3CrJ7BiK01hNr5yQbg1Ppz9Wb18FyhOd65MLmphhMvNAZGNzW18fMcIdM7YnvTF6K1MP%2B6vH38xEQpAehZe%2BXA8mt7%2FEHoA%2Bo%2BsjockIQK%2FhGbLD0zpjqLWEngn8XxWT9c9UjTbX2CHotlelVkgLCuz6iAv1a1kIb9gGnKBUjcAB7HZFwlVghC%2FrnnGqx%2FqrVLGu9B0aPQDT4PkGDQgU1a878SE6D3yRBJ1lqp64ZTMsC1p3ibZk8Qtrvg6N%2FWGkYuYoH%2BAtzAIby%2BctqvhU4HPnELzG0OVmF8K7VOa1oqxSpAvQFd4sqj6bHfcxcd1wvVqz6vUT7sY44Z5n%2F5KMYcwSFI5tVrxqMOfk1qUGOp4B85T13aDycd2vXm9dd7G0lFhJA3MEV8%2Fwxk89KzccV%2Fe6B%2BzeTDI8utqNQwJdQFzwtslD8O38wSA7ZD8sjrhP9ifVXK4TONWcW34ui5s0Aa2SXd4XQlHmoYhFdn8h55ovqpo3FNQOoFUJ9TeQK29tXh5gsmWOKAh3%2BrfbNUXYgxVNzkmmtb%2FBkOjfWRDRyOhe21ebtBej7Mp5il%2FeCCw%3D&amp;X-Amz-SignedHeaders=content-length%3Bcontent-type%3Bhost&amp;uploadId=pN8ZWQSSRiGpbk0rPZcRsOjyYFJF8JyKzkGiGqqrOEEviA825N8eQHuUiiUIPrSndk_B1PK3CwzBXXdQZOEJucKtSeHhX6i98Xcf1ewBBkFNItLK728Bnw0vinmtI4MyAr3R5pgWQ8k3GFN7KSuJ7XJZ2TVR9wWRHGlM9yhSgrbPZX48vOQooTtgQpXz475d
content-length:185
content-type:application/x-www-form-urlencoded
host:mybucket.s3.eu-central-1.amazonaws.com

content-length;content-type;host
UNSIGNED-PAYLOAD</CanonicalRequest>
  <CanonicalRequestBytes>50 4f 53 54 0a 2f 66 69 6c 65 0a 25 33 43 43 6f 6d 70 6c 65 74 65 4d 75 6c 74 69 70 61 72 74 55 70 6c 6f 61 64 25 32 30 78 6d 6c 6e 73 3d 25 32 32 68 74 74 70 25 33 41 25 32 46 25 32 46 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 25 32 46 64 6f 63 25 32 46 32 30 30 36 2d 30 33 2d 30 31 25 32 46 25 32 32 25 33 45 25 33 43 50 61 72 74 25 33 45 25 33 43 45 54 61 67 25 33 45 25 32 32 64 66 64 31 65 61 34 66 63 31 37 62 33 35 64 39 36 31 32 34 39 32 37 62 32 63 31 35 34 38 32 33 25 32 32 25 33 43 25 32 46 45 54 61 67 25 33 45 25 33 43 50 61 72 74 4e 75 6d 62 65 72 25 33 45 31 25 33 43 25 32 46 50 61 72 74 4e 75 6d 62 65 72 25 33 45 25 33 43 25 32 46 50 61 72 74 25 33 45 25 33 43 25 32 46 43 6f 6d 70 6c 65 74 65 4d 75 6c 74 69 70 61 72 74 55 70 6c 6f 61 64 25 33 45 26 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 53 49 41 34 34 4a 53 53 44 54 4e 4c 4e 4a 37 46 59 50 4b 25 32 46 32 30 32 33 30 37 31 37 25 32 46 65 75 2d 63 65 6e 74 72 61 6c 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 33 30 37 31 37 54 32 31 32 38 31 33 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 65 63 75 72 69 74 79 2d 54 6f 6b 65 6e 3d 49 51 6f 4a 62 33 4a 70 5a 32 6c 75 58 32 56 6a 45 4e 37 25 32 46 25 32 46 25 32 46 25 32 46 25 32 46 25 32 46 25 32 46 25 32 46 25 32 46 25 32 46 77 45 61 44 47 56 31 4c 57 4e 6c 62 6e 52 79 59 57 77 74 4d 53 4a 47 4d 45 51 43 49 46 56 47 59 53 78 4d 25 32 42 25 32 42 59 55 37 67 47 6c 67 58 74 49 67 6d 41 25 32 46 79 35 55 6d 6e 6f 6f 63 76 58 57 77 34 42 78 78 4b 36 50 55 41 69 41 37 31 5a 61 53 45 67 7a 6c 42 39 4d 6f 52 55 34 4b 77 53 6e 4a 6e 52 53 52 44 66 6b 49 51 51 32 43 71 6c 43 74 43 70 4d 25 32 46 25 32 46 79 71 78 41 67 68 6e 45 41 49 61 44 44 67 34 4e 54 51 77 4e 6a 45 33 4f 44 55 79 4d 69 49 4d 4e 49 35 64 46 70 79 35 47 37 4b 5a 6e 56 6e 63 4b 6f 34 43 37 58 25 32 42 25 32 46 6e 48 46 7a 33 43 72 4a 37 42 69 4b 30 31 68 4e 72 35 79 51 62 67 31 50 70 7a 39 57 62 31 38 46 79 68 4f 64 36 35 4d 4c 6d 70 68 68 4d 76 4e 41 5a 47 4e 7a 57 31 38 66 4d 63 49 64 4d 37 59 6e 76 54 46 36 4b 31 4d 50 25 32 42 36 76 48 33 38 78 45 51 70 41 65 68 5a 65 25 32 42 58 41 38 6d 74 37 25 32 46 45 48 6f 41 25 32 42 6f 25 32 42 73 6a 6f 63 6b 49 51 4b 25 32 46 68 47 62 4c 44 30 7a 70 6a 71 4c 57 45 6e 67 6e 38 58 78 57 54 39 63 39 55 6a 54 62 58 32 43 48 6f 74 6c 65 6c 56 6b 67 4c 43 75 7a 36 69 41 76 31 61 31 6b 49 62 39 67 47 6e 4b 42 55 6a 63 41 42 37 48 5a 46 77 6c 56 67 68 43 25 32 46 72 6e 6e 47 71 78 25 32 46 71 72 56 4c 47 75 39 42 30 61 50 51 44 54 34 50 6b 47 44 51 67 55 31 61 38 37 38 53 45 36 44 33 79 52 42 4a 31 6c 71 70 36 34 5a 54 4d 73 43 31 70 33 69 62 5a 6b 38 51 74 72 76 67 36 4e 25 32 46 57 47 6b 59 75 59 6f 48 25 32 42 41 74 7a 41 49 62 79 25 32 42 63 74 71 76 68 55 34 48 50 6e 45 4c 7a 47 30 4f 56 6d 46 38 4b 37 56 4f 61 31 6f 71 78 53 70 41 76 51 46 64 34 73 71 6a 36 62 48 66 63 78 63 64 31 77 76 56 71 7a 36 76 55 54 37 73 59 34 34 5a 35 6e 25 32 46 35 4b 4d 59 63 77 53 46 49 35 74 56 72 78 71 4d 4f 66 6b 31 71 55 47 4f 70 34 42 38 35 54 31 33 61 44 79 63 64 32 76 58 6d 39 64 64 37 47 30 6c 46 68 4a 41 33 4d 45 56 38 25 32 46 77 78 6b 38 39 4b 7a 63 63 56 25 32 46 65 36 42 25 32 42 7a 65 54 44 49 38 75 74 71 4e 51 77 4a 64 51 46 7a 77 74 73 6c 44 38 4f 33 38 77 53 41 37 5a 44 38 73 6a 72 68 50 39 69 66 56 58 4b 34 54 4f 4e 57 63 57 33 34 75 69 35 73 30 41 61 32 53 58 64 34 58 51 6c 48 6d 6f 59 68 46 64 6e 38 68 35 35 6f 76 71 70 6f 33 46 4e 51 4f 6f 46 55 4a 39 54 65 51 4b 32 39 74 58 68 35 67 73 6d 57 4f 4b 41 68 33 25 32 42 72 66 62 4e 55 58 59 67 78 56 4e 7a 6b 6d 6d 74 62 25 32 46 42 6b 4f 6a 66 57 52 44 52 79 4f 68 65 32 31 65 62 74 42 65 6a 37 4d 70 35 69 6c 25 32 46 65 43 43 77 25 33 44 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 25 33 42 63 6f 6e 74 65 6e 74 2d 74 79 70 65 25 33 42 68 6f 73 74 26 75 70 6c 6f 61 64 49 64 3d 70 4e 38 5a 57 51 53 53 52 69 47 70 62 6b 30 72 50 5a 63 52 73 4f 6a 79 59 46 4a 46 38 4a 79 4b 7a 6b 47 69 47 71 71 72 4f 45 45 76 69 41 38 32 35 4e 38 65 51 48 75 55 69 69 55 49 50 72 53 6e 64 6b 5f 42 31 50 4b 33 43 77 7a 42 58 58 64 51 5a 4f 45 4a 75 63 4b 74 53 65 48 68 58 36 69 39 38 58 63 66 31 65 77 42 42 6b 46 4e 49 74 4c 4b 37 32 38 42 6e 77 30 76 69 6e 6d 74 49 34 4d 79 41 72 33 52 35 70 67 57 51 38 6b 33 47 46 4e 37 4b 53 75 4a 37 58 4a 5a 32 54 56 52 39 77 57 52 48 47 6c 4d 39 79 68 53 67 72 62 50 5a 58 34 38 76 4f 51 6f 6f 54 74 67 51 70 58 7a 34 37 35 64 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 31 38 35 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 65 64 0a 68 6f 73 74 3a 69 38 36 37 37 36 31 2d 74 65 73 74 33 2e 73 33 2e 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3b 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3b 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
  <RequestId>R6YB1FTBBDXPTN1H</RequestId>
  <HostId>Vi5N1Gnj0hFP4fiaAzbrBJqo0xlKKkrIqDOcjmhV6tF16gZ3jX7Kg3QG5Yw9BIXN9vZu1Fib1Qeczs/Fv4oCqA==</HostId>
</Error>

As you can see, the content-type is set to application/x-www-form-urlencoded. I already tried with application/octet-stream and also forcing cUrl to send no content-type, but the result is always the same.

Reproduction Steps

I tried to make the code as simple and straightforward as possible, although still keeping it complete (so you can see that create-mpu and upload-part work, it's only complete-mpu that fails). You only need to fill up the configuration in method test and run the code to see the error.

package com.gmail.jeffcfbr;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.http.SdkHttpMethod;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest;
import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload;
import software.amazon.awssdk.services.s3.model.CompletedPart;
import software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest;
import software.amazon.awssdk.services.s3.model.UploadPartRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.CompleteMultipartUploadPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.CreateMultipartUploadPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedCompleteMultipartUploadRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedCreateMultipartUploadRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedUploadPartRequest;
import software.amazon.awssdk.services.s3.presigner.model.UploadPartPresignRequest;
import software.amazon.awssdk.utils.IoUtils;

import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamWriter;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.StringWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Map;

public class TestMPUWithPresignedURLsGH {

  private final XmlMapper xmlMapper = new XmlMapper();

  public static void main(final String[] args) throws IOException, XMLStreamException {
    new TestMPUWithPresignedURLsGH().test();
  }

  private void test() throws IOException, XMLStreamException {
    // FILL THE INFORMATION HERE
    final String accessKeyId = "";
    final String secretAccessKey = "";
    final Region region = null;
    final String bucket = "";
    final String key = "";

    this.putObjectMPU(region, accessKeyId, secretAccessKey, bucket, key);
  }

  private void putObjectMPU(final Region region, final String accessKeyId, final String secretAccessKey, final String bucket, final String key) throws IOException, XMLStreamException {
    // Create S3 presigner
    final AwsCredentialsProvider awsCredentialsProvider = () -> AwsBasicCredentials.create(accessKeyId, secretAccessKey);

    final S3Configuration s3Configuration = S3Configuration.builder()
            .pathStyleAccessEnabled(false)
            .build();

    final S3Presigner s3Presigner = S3Presigner.builder()
            .credentialsProvider(awsCredentialsProvider)
            .region(region)
            .serviceConfiguration(s3Configuration)
            .build();
    // -->

    // Create MPU
    System.out.printf("CreateMultipartUploadRequest: bucket=%s key=%s%n", bucket, key);
    final CreateMultipartUploadRequest createMultipartUploadRequest = CreateMultipartUploadRequest.builder()
            .bucket(bucket)
            .key(key)
            .contentType("application/octet-stream")
            .build();

    final CreateMultipartUploadPresignRequest createMultipartUploadPresignRequest = CreateMultipartUploadPresignRequest.builder()
            .signatureDuration(Duration.ofHours(1))
            .createMultipartUploadRequest(createMultipartUploadRequest)
            .build();

    final PresignedCreateMultipartUploadRequest presignedCreateMultipartUploadRequest = s3Presigner.presignCreateMultipartUpload(createMultipartUploadPresignRequest);
    final URL presignedCreateMultipartUploadRequestURL = presignedCreateMultipartUploadRequest.url();
    System.out.println("Create MPU URL: " + presignedCreateMultipartUploadRequestURL);

    final HttpURLResponse presignedCreateMultipartUploadRequestResponse = this.makeRequest(SdkHttpMethod.POST, presignedCreateMultipartUploadRequestURL, "application/octet-stream", /* data */ null);
    final JsonNode presignedCreateMultipartUploadRequestResponseXml = this.fromXml(presignedCreateMultipartUploadRequestResponse.getBody());
    final String multipartUploadId = presignedCreateMultipartUploadRequestResponseXml.get("UploadId").asText();

    System.out.println("MPU ID: " + multipartUploadId);
    // -->

    // MPU Part
    final int partNumber = 1;
    final String partContent = "AAA" + this.createLongString(1024 * 1024 * 5);
    final byte[] partContentBytes = partContent.getBytes(StandardCharsets.UTF_8);

    System.out.printf("UploadPartRequest #%d: bucket=%s key=%s upload-id=%s%n", partNumber, bucket, key, multipartUploadId);

    final UploadPartRequest uploadPartRequest = UploadPartRequest.builder()
            .bucket(bucket)
            .key(key)
            .uploadId(multipartUploadId)
            .partNumber(partNumber)
            .build();

    final UploadPartPresignRequest uploadPartPresignRequest = UploadPartPresignRequest.builder()
            .signatureDuration(Duration.ofHours(1))
            .uploadPartRequest(uploadPartRequest)
            .build();

    final PresignedUploadPartRequest presignedUploadPartRequest = s3Presigner.presignUploadPart(uploadPartPresignRequest);
    final URL presignedUploadPartRequestURL = presignedUploadPartRequest.url();
    System.out.printf("UploadPart #%d URL: %s%n", partNumber, presignedUploadPartRequestURL);

    final HttpURLResponse presignedUploadPartRequest1Response = this.makeRequest(SdkHttpMethod.PUT, presignedUploadPartRequestURL, /* contentType */ null, partContentBytes);
    final String eTag = presignedUploadPartRequest1Response.getHeaders().get("ETag").get(0);
    System.out.printf("UploadPart #%d ETag: %s%n", partNumber, eTag);

    final CompletedPart completedPart = CompletedPart.builder()
            .partNumber(partNumber)
            .eTag(eTag)
            .build();

    final String eTagPart = completedPart.eTag();
    // -->

    // Commit MPU
    final CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder()
            .parts(completedPart)
            .build();

    System.out.printf("CompleteMultipartUploadRequest: bucket=%s key=%s multipart-id=%s%n", bucket, key, multipartUploadId);
    final CompleteMultipartUploadRequest completeMultipartUploadRequest = CompleteMultipartUploadRequest.builder()
            .bucket(bucket)
            .key(key)
            .uploadId(multipartUploadId)
            .multipartUpload(completedMultipartUpload)
            .build();

    final CompleteMultipartUploadPresignRequest completeMultipartUploadPresignRequest = CompleteMultipartUploadPresignRequest.builder()
            .signatureDuration(Duration.ofHours(1))
            .completeMultipartUploadRequest(completeMultipartUploadRequest)
            .build();

    final PresignedCompleteMultipartUploadRequest presignedCompleteMultipartUploadRequest = s3Presigner.presignCompleteMultipartUpload(completeMultipartUploadPresignRequest);
    final URL presignedCompleteMultipartUploadRequestURL = presignedCompleteMultipartUploadRequest.url();
    System.out.printf("CompleteMultipartUpload URL: %s%n", presignedCompleteMultipartUploadRequestURL);

    final Map<Integer, String> parts = Collections.singletonMap(partNumber, eTagPart);
    final String completeMultipartRequestPayload = this.buildCompleteMultipartRequestPayload(parts);
    final byte[] completeMultipartRequestPayloadBytes = completeMultipartRequestPayload.getBytes(StandardCharsets.UTF_8);

    System.out.println("Request payload: " + new String(completeMultipartRequestPayloadBytes, StandardCharsets.UTF_8));

    final HttpURLResponse completeMultipartUploadResponse = this.makeRequest(SdkHttpMethod.POST, presignedCompleteMultipartUploadRequestURL, /* contentType */ "application/x-www-form-urlencoded", completeMultipartRequestPayloadBytes);

    System.out.println("Status Code: " + completeMultipartUploadResponse.getStatusCode());
    System.out.println("Response headers: " + completeMultipartUploadResponse.getHeaders());
    System.out.println("Response body: " + completeMultipartUploadResponse.getBody());
    // -->
  }

  private String buildCompleteMultipartRequestPayload(final Map<Integer, String> parts) throws XMLStreamException {
    final StringWriter stringWriter = new StringWriter();
    final XMLOutputFactory xmlOutputFactory = XMLOutputFactory.newFactory();
    final XMLStreamWriter xmlWriter = xmlOutputFactory.createXMLStreamWriter(stringWriter);

    xmlWriter.writeStartDocument();
    xmlWriter.writeStartElement("CompleteMultipartUpload");
    xmlWriter.writeDefaultNamespace("http://s3.amazonaws.com/doc/2006-03-01/");

    for (final Map.Entry<Integer, String> part : parts.entrySet()) {
      xmlWriter.writeStartElement("Part");

      xmlWriter.writeStartElement("ETag");
      xmlWriter.writeCharacters(part.getValue());
      xmlWriter.writeEndElement();

      xmlWriter.writeStartElement("PartNumber");
      xmlWriter.writeCharacters(part.getKey().toString());
      xmlWriter.writeEndElement();

      xmlWriter.writeEndElement();
    }

    xmlWriter.writeEndElement();
    xmlWriter.writeEndDocument();

    return stringWriter.toString().replace("<?xml version='1.0' encoding='UTF-8'?>", "");
  }

  private String createLongString(final int length) {
    final char[] randomChars = new char[length];
    Arrays.fill(randomChars, 'z');

    return new String(randomChars);
  }

  private JsonNode fromXml(final String xml) throws JsonProcessingException {
    return this.xmlMapper.readTree(xml);
  }

  private HttpURLResponse makeRequest(final SdkHttpMethod httpMethod, final URL url, final String contentType, final byte[] data) throws IOException {
    System.out.println("Making request to: " + url.toString());

    final HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
    httpURLConnection.setRequestMethod(httpMethod.name());
    httpURLConnection.setDefaultUseCaches(false);

    if (contentType != null) {
      httpURLConnection.setRequestProperty("Content-Type", contentType);
    }

    if (data != null) {
      httpURLConnection.setDoOutput(true);

      try (final InputStream requestPayloadInputStream = new ByteArrayInputStream(data); final OutputStream httpOutputStream = httpURLConnection.getOutputStream()) {
        IoUtils.copy(requestPayloadInputStream, httpOutputStream);
      }
    }

    return new HttpURLResponse(httpURLConnection);
  }

  private static class HttpURLResponse {

    private final int statusCode;
    private final Map<String, List<String>> headers;
    private final String body;

    public HttpURLResponse(final HttpURLConnection httpURLConnection) throws IOException {
      this.statusCode = httpURLConnection.getResponseCode();
      this.headers = httpURLConnection.getHeaderFields();

      if (this.statusCode == HttpURLConnection.HTTP_OK) {
        try (final InputStream in = httpURLConnection.getInputStream(); final ByteArrayOutputStream out = new ByteArrayOutputStream()) {
          IoUtils.copy(in, out);
          final byte[] bodyBytes = out.toByteArray();
          this.body = new String(bodyBytes, StandardCharsets.UTF_8);
        }
      } else {
        try (final InputStream in = httpURLConnection.getErrorStream(); final ByteArrayOutputStream out = new ByteArrayOutputStream()) {
          IoUtils.copy(in, out);
          final byte[] bodyBytes = out.toByteArray();
          this.body = new String(bodyBytes, StandardCharsets.UTF_8);
        }
      }
    }

    public int getStatusCode() {
      return this.statusCode;
    }

    public Map<String, List<String>> getHeaders() {
      return this.headers;
    }

    public String getBody() {
      return this.body;
    }

  }

}

Possible Solution

No response

Additional Information/Context

It's important to mention that I am able to make the MPU with no presigned URLs. i.e, I have confirmed that the credentials are valid, and that they have permission to s3:PutObject, and that the API works correctly for completing an MPU. The issue only arises when using presigned URLs.

It's also relevant to mention that if I take the complete-mpu XML payload, convert it to JSON, and pass it to the CLI, the MPU gets completed successfully. That confirms that create-mpu and upload-part steps are working as expected.

Output of running the code.

CreateMultipartUploadRequest: bucket=mybucket key=file
Create MPU URL: https://mybucket.s3.eu-central-1.amazonaws.com/file?uploads&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230717T214932Z&X-Amz-SignedHeaders=content-type%3Bhost&X-Amz-Expires=3599&X-Amz-Credential=[REDACTED]%2F20230717%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=3fb3df7bf6dbfa2f19266a769a28952ebb8e66432de4e8b79bdc80b76aa270f6
Making request to: https://mybucket.s3.eu-central-1.amazonaws.com/file?uploads&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230717T214932Z&X-Amz-SignedHeaders=content-type%3Bhost&X-Amz-Expires=3599&X-Amz-Credential=[REDACTED]%2F20230717%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=3fb3df7bf6dbfa2f19266a769a28952ebb8e66432de4e8b79bdc80b76aa270f6
MPU ID: oS15xEyI06lIhvR6BCGdiNlolPdwwduYcCfHacqwBSJ8N.amJ8HVIEqrWM7FS8CoDCtRxJ9hizyydu5pXeRJZg8vHcT14YDaNy.z_Upo6BWgdU9_VUIl7y44mzFTt8Vq
UploadPartRequest #1: bucket=mybucket key=file upload-id=oS15xEyI06lIhvR6BCGdiNlolPdwwduYcCfHacqwBSJ8N.amJ8HVIEqrWM7FS8CoDCtRxJ9hizyydu5pXeRJZg8vHcT14YDaNy.z_Upo6BWgdU9_VUIl7y44mzFTt8Vq
UploadPart #1 URL: https://mybucket.s3.eu-central-1.amazonaws.com/file?partNumber=1&uploadId=oS15xEyI06lIhvR6BCGdiNlolPdwwduYcCfHacqwBSJ8N.amJ8HVIEqrWM7FS8CoDCtRxJ9hizyydu5pXeRJZg8vHcT14YDaNy.z_Upo6BWgdU9_VUIl7y44mzFTt8Vq&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230717T214934Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=[REDACTED]%2F20230717%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=[REDACTED]
Making request to: https://mybucket.s3.eu-central-1.amazonaws.com/file?partNumber=1&uploadId=oS15xEyI06lIhvR6BCGdiNlolPdwwduYcCfHacqwBSJ8N.amJ8HVIEqrWM7FS8CoDCtRxJ9hizyydu5pXeRJZg8vHcT14YDaNy.z_Upo6BWgdU9_VUIl7y44mzFTt8Vq&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230717T214934Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=[REDACTED]%2F20230717%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=[REDACTED]
UploadPart #1 ETag: "dfd1ea4fc17b35d96124927b2c154823"
CompleteMultipartUploadRequest: bucket=mybucket key=file multipart-id=oS15xEyI06lIhvR6BCGdiNlolPdwwduYcCfHacqwBSJ8N.amJ8HVIEqrWM7FS8CoDCtRxJ9hizyydu5pXeRJZg8vHcT14YDaNy.z_Upo6BWgdU9_VUIl7y44mzFTt8Vq
CompleteMultipartUpload URL: https://mybucket.s3.eu-central-1.amazonaws.com/file?uploadId=oS15xEyI06lIhvR6BCGdiNlolPdwwduYcCfHacqwBSJ8N.amJ8HVIEqrWM7FS8CoDCtRxJ9hizyydu5pXeRJZg8vHcT14YDaNy.z_Upo6BWgdU9_VUIl7y44mzFTt8Vq&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230717T214937Z&X-Amz-SignedHeaders=content-length%3Bcontent-type%3Bhost&X-Amz-Expires=3600&X-Amz-Credential=[REDACTED]%2F20230717%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=652e300e6127486866bf341550e8ef68e631ea92a45035975729899494127fdc
Request payload: <CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Part><ETag>"dfd1ea4fc17b35d96124927b2c154823"</ETag><PartNumber>1</PartNumber></Part></CompleteMultipartUpload>
Making request to: https://mybucket.s3.eu-central-1.amazonaws.com/file?uploadId=oS15xEyI06lIhvR6BCGdiNlolPdwwduYcCfHacqwBSJ8N.amJ8HVIEqrWM7FS8CoDCtRxJ9hizyydu5pXeRJZg8vHcT14YDaNy.z_Upo6BWgdU9_VUIl7y44mzFTt8Vq&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230717T214937Z&X-Amz-SignedHeaders=content-length%3Bcontent-type%3Bhost&X-Amz-Expires=3600&X-Amz-Credential=[REDACTED]%2F20230717%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=652e300e6127486866bf341550e8ef68e631ea92a45035975729899494127fdc
Status Code: 403
Response headers: {Transfer-Encoding=[chunked], null=[HTTP/1.1 403 Forbidden], Server=[AmazonS3], x-amz-request-id=[CDQJMN0H6N6Q249W], x-amz-id-2=[4di4QGvHOThVuJ3/cSZV44aCc2q4wfl5vKzcsmQVeCbbGQPBzUyGiv2bfC3JtlLzVDxDDwdYmBxWs7pTMw9SeA==], Date=[Mon, 17 Jul 2023 21:49:36 GMT], Content-Type=[application/xml]}
Response body: <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>[REDACTED]</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20230717T214937Z
20230717/eu-central-1/s3/aws4_request
0a017e51af84f8269cbf16b38d97120234564a0e1c4bec128e9e5b7e3961a3a9</StringToSign><SignatureProvided>652e300e6127486866bf341550e8ef68e631ea92a45035975729899494127fdc</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 33 30 37 31 37 54 32 31 34 39 33 37 5a 0a 32 30 32 33 30 37 31 37 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 30 61 30 31 37 65 35 31 61 66 38 34 66 38 32 36 39 63 62 66 31 36 62 33 38 64 39 37 31 32 30 32 33 34 35 36 34 61 30 65 31 63 34 62 65 63 31 32 38 65 39 65 35 62 37 65 33 39 36 31 61 33 61 39</StringToSignBytes><CanonicalRequest>POST
/file
%3CCompleteMultipartUpload%20xmlns=%22http%3A%2F%2Fs3.amazonaws.com%2Fdoc%2F2006-03-01%2F%22%3E%3CPart%3E%3CETag%3E%22dfd1ea4fc17b35d96124927b2c154823%22%3C%2FETag%3E%3CPartNumber%3E1%3C%2FPartNumber%3E%3C%2FPart%3E%3C%2FCompleteMultipartUpload%3E&amp;X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=[REDACTED]%2F20230717%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20230717T214937Z&amp;X-Amz-Expires=3600&amp;X-Amz-SignedHeaders=content-length%3Bcontent-type%3Bhost&amp;uploadId=oS15xEyI06lIhvR6BCGdiNlolPdwwduYcCfHacqwBSJ8N.amJ8HVIEqrWM7FS8CoDCtRxJ9hizyydu5pXeRJZg8vHcT14YDaNy.z_Upo6BWgdU9_VUIl7y44mzFTt8Vq
content-length:185
content-type:application/x-www-form-urlencoded
host:mybucket.s3.eu-central-1.amazonaws.com

content-length;content-type;host
UNSIGNED-PAYLOAD</CanonicalRequest><CanonicalRequestBytes>50 4f 53 54 0a 2f 66 69 6c 65 0a 25 33 43 43 6f 6d 70 6c 65 74 65 4d 75 6c 74 69 70 61 72 74 55 70 6c 6f 61 64 25 32 30 78 6d 6c 6e 73 3d 25 32 32 68 74 74 70 25 33 41 25 32 46 25 32 46 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 25 32 46 64 6f 63 25 32 46 32 30 30 36 2d 30 33 2d 30 31 25 32 46 25 32 32 25 33 45 25 33 43 50 61 72 74 25 33 45 25 33 43 45 54 61 67 25 33 45 25 32 32 64 66 64 31 65 61 34 66 63 31 37 62 33 35 64 39 36 31 32 34 39 32 37 62 32 63 31 35 34 38 32 33 25 32 32 25 33 43 25 32 46 45 54 61 67 25 33 45 25 33 43 50 61 72 74 4e 75 6d 62 65 72 25 33 45 31 25 33 43 25 32 46 50 61 72 74 4e 75 6d 62 65 72 25 33 45 25 33 43 25 32 46 50 61 72 74 25 33 45 25 33 43 25 32 46 43 6f 6d 70 6c 65 74 65 4d 75 6c 74 69 70 61 72 74 55 70 6c 6f 61 64 25 33 45 26 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 34 34 4a 53 53 44 54 4e 42 5a 4a 4e 48 35 48 55 25 32 46 32 30 32 33 30 37 31 37 25 32 46 65 75 2d 63 65 6e 74 72 61 6c 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 33 30 37 31 37 54 32 31 34 39 33 37 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 25 33 42 63 6f 6e 74 65 6e 74 2d 74 79 70 65 25 33 42 68 6f 73 74 26 75 70 6c 6f 61 64 49 64 3d 6f 53 31 35 78 45 79 49 30 36 6c 49 68 76 52 36 42 43 47 64 69 4e 6c 6f 6c 50 64 77 77 64 75 59 63 43 66 48 61 63 71 77 42 53 4a 38 4e 2e 61 6d 4a 38 48 56 49 45 71 72 57 4d 37 46 53 38 43 6f 44 43 74 52 78 4a 39 68 69 7a 79 79 64 75 35 70 58 65 52 4a 5a 67 38 76 48 63 54 31 34 59 44 61 4e 79 2e 7a 5f 55 70 6f 36 42 57 67 64 55 39 5f 56 55 49 6c 37 79 34 34 6d 7a 46 54 74 38 56 71 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 31 38 35 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 65 64 0a 68 6f 73 74 3a 69 38 36 37 37 36 31 2d 74 65 73 74 33 2e 73 33 2e 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3b 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3b 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes><RequestId>CDQJMN0H6N6Q249W</RequestId><HostId>4di4QGvHOThVuJ3/cSZV44aCc2q4wfl5vKzcsmQVeCbbGQPBzUyGiv2bfC3JtlLzVDxDDwdYmBxWs7pTMw9SeA==</HostId></Error>

AWS Java SDK version used

2.20.103

JDK version used

1.8.0_351

Operating System and version

macOS 13.4.1

jeffcfbr avatar Jul 17 '23 22:07 jeffcfbr

Here you can see a very similar code that makes the same MPU, except it doesn't use presigned URLs. I ran this code with the very same credentials, bucket, and object key, and this works. i.e., the issue is specifically with the use of presigned URLs.

TestMPUGH.java.zip

jeffcfbr avatar Jul 17 '23 22:07 jeffcfbr

Attaching the POM file. pom.xml.zip

jeffcfbr avatar Jul 24 '23 21:07 jeffcfbr

Replace the parameter --data in the curl command with --data-raw, and it can run successfully on my side

notnonot avatar Aug 30 '23 06:08 notnonot

-- Data in curl Cmd aside to Replace -- Data--Raw its work maybe

ujjwalshriv3 avatar Sep 01 '23 12:09 ujjwalshriv3

I am seeing the same issue with presignedCompletedMultipartUpload URL. But the same request works from java SDK's s3Client.completeMultipartUpload API

Error on trying to POST with presignedCompletedMultipartUpload URL, with the part numbers and respective ETag info: <Error> <Code>SignatureDoesNotMatch</Code> <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message> ....

AWS Java SDK version used 2.20.56

JDK version used openjdk version "11.0.20"

Operating System and version macOS 13.4.1

nithinks21 avatar Sep 29 '23 15:09 nithinks21

I'm trying with a Java-based HTTP request as well as with cUrl. Both failing.

Didn't work with --data-raw for me.

@notnonot Were you able to make it work? Please, would you kindly share your commands? From the MPU creation, to the parts upload, to the complete MPU request?

jeffcfbr avatar Nov 16 '23 20:11 jeffcfbr

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

github-actions[bot] avatar Mar 18 '24 21:03 github-actions[bot]