S3Mock icon indicating copy to clipboard operation
S3Mock copied to clipboard

Published 20 hours ago • verified publisher icon adobe

Content-MD5 does not match object md5

Open 2mars opened this issue 9 months ago • 4 comments

With 3.7.3 version my code now fails when I try to put file using amazon s3 client (putObject).

I have an exception (that I haven't before) :

12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> PUT /test-baguette/sentinel_epsg32720_classified.tiff HTTP/1.1 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Host: localhost:42908 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> amz-sdk-invocation-id: 7401ca2c-2c72-648b-709f-f63902fead17 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> amz-sdk-request: attempt=1;max=4 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> amz-sdk-retry: 0/0/500 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=foo/20240506/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=f226c1fd2282d702057f66e67d3d1bf55fdd4db0891faa7d14adc5037648fd97 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Content-MD5: h5qk8jrzBOtj94WoBhDShw== 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Content-Type: image/tiff 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> User-Agent: aws-sdk-java/1.12.712 Mac_OS_X/14.4.1 OpenJDK_64-Bit_Server_VM/17.0.2+8-86 java/17.0.2 vendor/Oracle_Corporation cfg/retry-mode/legacy cfg/auth-source#unknown 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> X-Amz-Date: 20240506T103749Z 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> x-amz-decoded-content-length: 18636 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Content-Length: 18811 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Connection: Keep-Alive 12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Expect: 100-continue 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "PUT /test-baguette/sentinel_epsg32720_classified.tiff HTTP/1.1[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Host: localhost:42908[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "amz-sdk-invocation-id: 7401ca2c-2c72-648b-709f-f63902fead17[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "amz-sdk-request: attempt=1;max=4[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=foo/20240506/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=f226c1fd2282d702057f66e67d3d1bf55fdd4db0891faa7d14adc5037648fd97[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Content-MD5: h5qk8jrzBOtj94WoBhDShw==[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Content-Type: image/tiff[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "User-Agent: aws-sdk-java/1.12.712 Mac_OS_X/14.4.1 OpenJDK_64-Bit_Server_VM/17.0.2+8-86 java/17.0.2 vendor/Oracle_Corporation cfg/retry-mode/legacy cfg/auth-source#unknown[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "X-Amz-Date: 20240506T103749Z[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "x-amz-decoded-content-length: 18636[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Content-Length: 18811[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Expect: 100-continue[\r][\n]" 12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "[\r][\n]" 12:37:49.900 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]" 12:37:49.900 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "[\r][\n]" 12:37:49.901 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << HTTP/1.1 100 Continue 12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "48cc;chunk-signature=ee792a1a7e167e4fa48767766c31422d2f7009e88217189abe8e6b75e9996fb2[\r][\n]" 12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "II*[0x0][0x8][0x0][0x0][0x0][0x11][0x0][0x0][0x1][0x3][0x0][0x1][0x0][0x0][0x0]B[0x0][0x0][0x0][0x1][0x1][0x3][0x0][0x1][0x0][0x0][0x0]D[0x0][0x0][0x0][0x2][0x1][0x3][0x0][0x1][0x0][0x0][0x0] [0x0][0x0][0x0][0x3][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1][0x0][0x0][0x0][0x6][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1][0x0][0x0][0x0][0x11][0x1][0x4][0x0][0x3][0x0][0x0][0x0][0xe6][0x0][0x0][0x0][0x15][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1][0x0][0x0][0x0][0x16][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1f][0x0][0x0][0x0][0x17][0x1][0x4][0x0][0x3][0x0][0x0][0x0][0xda][0x0][0x0][0x0][0x1c][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1][0x0][0x0][0x0]S[0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x2][0x0][0x0][0x0][0xe][0x83][0xc][0x0][0x3][0x0][0x0][0x0][0x6][0x2][0x0][0x0][0x82][0x84][0xc][0x0][0x6][0x0][0x0][0x0][0x1e][0x2][0x0][0x0][0xaf][0x87][0x3][0x0] [0x0][0x0][0x0]N[0x2][0x0][0x0][0xb1][0x87][0x2][0x0][0x1e][0x0][0x0][0x0][0x8e][0x2][0x0][0x0][0x80][0xa4][0x2][0x0][0x14][0x1][0x0][0x0][0xf2][0x0][0x0][0x0][0x81][0xa4][0x2][0x0][0x3][0x0][0x0][0x0]-1[0x0][0x0][0x0][0x0][0x0][0x0][0xf8][0x1f][0x0][0x0][0xf8][0x1f][0x0][0x0]0[0x6][0x0][0x0][0xac][0x2][0x0][0x0][0xa4]"[0x0][0x0][0x9c]B[0x0][0x0]<GDALMetadata>[\n]" 12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> " <Item name="STATISTICS_MAXIMUM" sample="0">2</Item>[\n]" 12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> " <Item name="STATISTICS_MEAN" sample="0">-0.41644385026738</Item>[\n]" 12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> " <Item name="STATISTICS_MINIMUM" sample="0">-1</Item>[\n]" 12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> " <Item name="STATISTICS_STDDEV" sample="0">0.93894098334153</Item>[\n]" 12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "</GDALMetadata>[\n]" 12:37:49.904 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "[0x0][0x0][0x0][0x0][0x0][0x0][0x0]$@[0x0][0x0] ..." 12:37:49.906 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "[0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0] ... [\r][\n]" 12:37:49.906 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "0;chunk-signature=804a6f3d7ff45f1bc11eec20f425c5fdb9d86609a19744aacf1c0d6a300a1f4b[\r][\n]" 12:37:49.906 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Server: Jetty(12.0.8)[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Date: Mon, 06 May 2024 10:37:49 GMT[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Vary: Origin[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Vary: Access-Control-Request-Method[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Vary: Access-Control-Request-Headers[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Content-Type: application/xml[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "84[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "<Error><Code>BadRequest</Code><Message>Content-MD5 does not match object md5</Message></Error>" 12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << HTTP/1.1 400 Bad Request 12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Server: Jetty(12.0.8) 12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Date: Mon, 06 May 2024 10:37:49 GMT 12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Vary: Origin 12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Vary: Access-Control-Request-Method 12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Vary: Access-Control-Request-Headers 12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Content-Type: application/xml 12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Transfer-Encoding: chunked 12:37:49.969 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Connection can be kept alive for 60000 MILLISECONDS 12:37:49.969 [main] DEBUG com.amazonaws.retry.ClockSkewAdjuster -- Reported server date (from 'Date' header): Mon, 06 May 2024 10:37:49 GMT 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "0[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "[\r][\n]" 12:37:49.969 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection [id: 0][route: {}->http://localhost:42908] can be kept alive for 60.0 seconds 12:37:49.969 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-0: set socket timeout to 0 12:37:49.969 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection released: [id: 0][route: {}->http://localhost:42908][total available: 1; route allocated: 1 of 50; total allocated: 1 of 50] 12:37:49.970 [main] DEBUG com.amazonaws.request -- Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Content-MD5 does not match object md5 (Service: Amazon S3; Status Code: 400; Error Code: BadRequest; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null 12:37:49.970 [main] DEBUG com.amazonaws.auth.AwsChunkedEncodingInputStream -- AwsChunkedEncodingInputStream reset (will reset the wrapped stream because it is mark-supported).

com.amazonaws.services.s3.model.AmazonS3Exception: Content-MD5 does not match object md5 (Service: Amazon S3; Status Code: 400; Error Code: BadRequest; Request ID: null; S3 Extended Request ID: null; Proxy: null) , S3 Extended Request ID: null at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1880) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5558) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5505) at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:423) at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6639) at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1892) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1852)

2mars avatar May 06 '24 10:05 2mars

@2mars this is probably related to #1840

Do you have a working test for me to reproduce this?

afranken avatar May 06 '24 11:05 afranken 

@Test
void fail_md5() throws URISyntaxException {
    AmazonS3 s3client = AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("foo", "bar")))
            .withEndpointConfiguration(new EndpointConfiguration("http://localhost:42908", "eu-central-1"))
            .enablePathStyleAccess().build();
    Bucket someTestBucket = s3client.createBucket("test-baguette");
    File someGeotiff = new File(this.getClass().getResource("/sentinel_epsg32720_classified.tiff").toURI());
    s3client.putObject(new PutObjectRequest(someTestBucket.getName(), someGeotiff.getName(), someGeotiff));
    assertThat(s3client.getObject(new GetObjectRequest(someTestBucket.getName(), someGeotiff.getName()))).isNotNull();
}

2mars avatar May 06 '24 12:05 2mars

thanks!

Looks like this only happens when using http as a connection method (which results in signed chunked uploads) in connection to uploading binary files like images. In our Integration-Tests and unit-tests we're using text files for testing. The Integration-Tests use https as a connection method with few exceptions.

That's why this issue was not caught during the refactorings I made for 3.7.3. I'll look into a solution.

afranken avatar May 06 '24 21:05 afranken

I'm having the same error with HTTP, however I'm uploading a plaintext JSON file. File size is ~3.5k, using a pretty vanilla container instance.

jonasholtkamp avatar May 08 '24 08:05 jonasholtkamp

@jonasholtkamp @2mars I released 3.8.0 earlier today which should fix the problems in most cases. Only the combination http / sync / sha256 does not work, at least with the AWS v2 SDK for Java.

afranken avatar May 12 '24 19:05 afranken

Upgraded to 3.8.0, everything works like it did in 3.7.2, thank you.

One extra question, might be a little offtopic: How does one set up the HTTPS client with S3Mock? The README mentions port 9191. If I shoot HTTPS request against that port, I get these errors:

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I suppose I need to trust a certificate first? Do I need to generate, self-sign and install one with the mock? Some guidance would be appreciated.

jonasholtkamp avatar May 13 '24 06:05 jonasholtkamp

@jonasholtkamp S3Mock bring its own, non-compliant SSL certificate. The HttpClient used by the AWS SDK can be configured to ignore that. The integration-test module contains setups for a number of HttpClients, for example: https://github.com/adobe/S3Mock/blob/main/integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/S3TestBase.kt#L156

If you run S3Mock in Docker, you could bring your own, valid SSL certificate, see #281

I'll see if I can add something to the README.md about using SSL in a later release.

afranken avatar May 13 '24 09:05 afranken