Multipart S3 uploads - composite checksums are not given the -X suffix

[sophieilangolyon1-nhs]

As per the discussion referenced below, composite checksums when generated by AWS S3 are given an -X suffix to distinguish them from full object checksums. Moto does not do this, so composite checksums present as full object checksums which causes a problem when retrieving the object using botocore (which attempts to validate the checksum against a full object checksum, inevitably failing validation).

The attached python code can be run against AWS (with correct permissions) or against moto (with the 'mock' argument) to demonstrate the difference of behaviour. Note the two ExtraArgs options at line 27/28 which can be used to direct AWS towards composite or full object checksum generation.

reproduce-checksum-generation-problem.txt

Originally posted by @sophieilangolyon1-nhs in https://github.com/getmoto/moto/discussions/8760

[bblommers]

Hi @sophieilangolyon1-nhs, thanks for the detailed reproducer, and welcome to Moto! I'll mark it as a bug.

If this is something you want to work on yourself, PR's are always welcome! The documentation should have everything you need to get started: https://docs.getmoto.org/en/latest/docs/contributing/index.html

[mfulgo]

Just to make this issue a bit more findable... The error I receive when attempting to read the object is similar to the following:

self = <botocore.httpchecksum.StreamingChecksumBody object at 0xffff9464bee0>

    def _validate_checksum(self):
        if self._checksum.digest() != base64.b64decode(self._expected):
            error_msg = (
                f"Expected checksum {self._expected} did not match calculated "
                f"checksum: {self._checksum.b64digest()}"
            )
>           raise FlexibleChecksumError(error_msg=error_msg)
E           botocore.exceptions.FlexibleChecksumError: Expected checksum TeqlgA== did not match calculated checksum: mhURAw==

To work around this, I'm using ChecksumMode="DISABLED" in my tests to avoid the error. For example:

content = s3_client.get_object(
    Bucket="test-bucket",
    Key="test-key1",
    ChecksumMode="DISABLED",
)["Body"].read()

[lbesnard]

For Ref, I encountered the same issue when updating poetry dependencies in my package. I found that since Moto 5.1.2 my unittest are failing with the same type of msg

botocore.exceptions.FlexibleChecksumError: Expected checksum TeqlgA== did not match calculated checksum: mhURAw==