Error when putting object in bucket : MD5 does not match etag when using Java AWS SDK

[StefH]

I'm using AWS S3 SDK to put a file into this fake s3.

Exception in thread "main" com.amazonaws.AmazonClientException: Unable to verify integrity of data upload. Client calculated content hash (contentMD5: rtGkDmuy7nJ1ms6ZoEyrKQ== in base 64) didn't match hash (etag: b348ec55910b6836d3f2bd8e69f484f0 in hex) calculated by Amazon S3. You may need to delete the data stored in Amazon S3. (metadata.contentMD5: null, md5DigestStream: com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream@78aea4b9, bucketName: hello-world, key: file_636120647747450000.txt) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1609) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:3147) at xxx.aws.s3.S3Context.putObjectAsString(S3Context.java:26)

[l15k4]

I think it's this https://github.com/jubos/fake-s3/issues/107

[l15k4]

I investigated further and the last version of aws-java-sdk-s3 that fake-s3 works with is 1.11.15 ... A breaking change must have been introduced in 1.11.16

[cescude]

FWIW, it looks like the Amazon client uses chunked encoding by default now (rather than sending the raw bytes), and fake-s3 ends up calculating the md5 on encoded payload, not the raw data.

You can check by navigating to where fakes3 stores the data, running md5sum content, and comparing that to the md5sum of the original file.

A quick workaround is to initialize the AmazonS3Client with chunked-mode disabled, i.e.:

s3client.setS3ClientOptions(
  S3ClientOptions.builder
    .setPathStyleAccess(true)
    .disableChunkedEncoding.build)

When I do this locally, everything works fine.

(I'm running with the AWS SDK version 1.11.41)

[kknd22]

can we have a cleaner fix other than this flag?

[kknd22]

this make the production code look like:

if (USING_FASKES3)
  AmazonS3ClientBuilder.standard().disableChunkedEncoding
else 
    AmazonS3ClientBuilder.standard()  

should have better solution?

-cl

[jubos]

I agree that this is a pretty clunky solution. Will investigate a solution and see if it is easy to get on the roadmap.

[jensmith-work]

the work-around works but as kknd22 commented, I would also like a cleaner solution