S3 Client delete_objects() missing required Content-MD5 header with FIPS mode enabled

[rickatnight11]

Affected Version(s)

From what I tested:

boto3 (1.17.13)
botocore (1.20.13)

Describe the bug Issuing a properly-formatted delete_objects() call returns:

botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the DeleteObjects operation: Missing required header for this request: Content-MD5

It looks related to https://github.com/boto/botocore/issues/1302 filed back in 2017, but perhaps there's been a regression in this particular code-path (or a change in the API).

Steps to reproduce

1. Spin up an S3 bucket and fill with some sample objects
2. Run code snippit:

import boto3
BUCKET='<bucketname>'
s3_client = boto3.client('s3')
object_response_paginator = s3_client.get_paginator('list_object_versions')
object_version_list = []
for object_response_itr in object_response_paginator.paginate(Bucket=BUCKET):
    if 'Versions' in object_response_itr:
        for version in object_response_itr['Versions']:
            object_version_list.append({'Key': version['Key'],
                                 'VersionId': version['VersionId']})
for i in range(0, len(object_version_list), 1000):
    response = s3_client.delete_objects(
        Bucket=bucket_name,
        Delete={
            'Objects': object_version_list[i:i+1000],
            'Quiet': True
        }
    )

3. Observe exception:

Traceback (most recent call last):
  File "bucket_delete.py", line 17, in <module>
    'Quiet': True
  File "/home/maintuser/.local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/maintuser/.local/lib/python3.6/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the DeleteObjects operation:
 Missing required header for this request: Content-MD5

Expected behavior* Objects (including versions, if a versioned bucket) are deleted.

Debug logs

2021-02-23 15:38:14,471 - botocore.hooks - DEBUG - Event before-parameter-build.s3.DeleteObjects: calling handler <function validate_bucket_name at 0x7f39d40e0510>
2021-02-23 15:38:14,471 - botocore.hooks - DEBUG - Event before-parameter-build.s3.DeleteObjects: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f39d263e9e8>>
2021-02-23 15:38:14,471 - botocore.hooks - DEBUG - Event before-parameter-build.s3.DeleteObjects: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f39d263eda0>>
2021-02-23 15:38:14,471 - botocore.hooks - DEBUG - Event before-parameter-build.s3.DeleteObjects: calling handler <function generate_idempotent_uuid at 0x7f39d40e0378>
2021-02-23 15:38:14,472 - botocore.hooks - DEBUG - Event before-call.s3.DeleteObjects: calling handler <function add_expect_header at 0x7f39d40e0840>
2021-02-23 15:38:14,472 - botocore.hooks - DEBUG - Event before-call.s3.DeleteObjects: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f39d263e9e8>>
2021-02-23 15:38:14,472 - botocore.hooks - DEBUG - Event before-call.s3.DeleteObjects: calling handler <function inject_api_version_header_if_needed at 0x7f39d40e2ae8>
2021-02-23 15:38:14,472 - botocore.endpoint - DEBUG - Making request for OperationModel(name=DeleteObjects) with params: {'url_path': '/example-test-bucket?delete', 'query_string': {}, 'method': 'POST', 'headers': {'User-Agent': 'Boto3/1.17.13 Python/3.6.8 Linux/3.10.0-1160.15.2.el7.x86_64 Botocore/1.20.13'}, 'body': b'<Delete xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Object><Key>test_file_10</Key><VersionId>UfYcpa7soJwRaP7fPyReSG50MJmmMhqZ</VersionId></Object><Quiet>true</Quiet></Delete>', 'url': 'https://s3.amazonaws.com/example-test-bucket?delete', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f39d263e6a0>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'example-test-bucket'}}}
2021-02-23 15:38:14,472 - botocore.hooks - DEBUG - Event request-created.s3.DeleteObjects: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f39d263e668>>
2021-02-23 15:38:14,472 - botocore.hooks - DEBUG - Event choose-signer.s3.DeleteObjects: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f39d2dd3fd0>>
2021-02-23 15:38:14,472 - botocore.hooks - DEBUG - Event choose-signer.s3.DeleteObjects: calling handler <function set_operation_specific_signer at 0x7f39d40e0268>
2021-02-23 15:38:14,472 - botocore.hooks - DEBUG - Event before-sign.s3.DeleteObjects: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f39d263ee10>>
2021-02-23 15:38:14,472 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/example-test-bucket?delete
2021-02-23 15:38:14,472 - botocore.utils - DEBUG - URI updated to: https://example-test-bucket.s3.amazonaws.com/?delete
2021-02-23 15:38:14,473 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-02-23 15:38:14,473 - botocore.auth - DEBUG - CanonicalRequest:
POST
/
delete=
host:example-test-bucket.s3.amazonaws.com
x-amz-content-sha256:2686acdcb1fdcf33a602537d0383bc85a4ccbca860cbef71825e616c5ddc753d
x-amz-date:20210223T153814Z

host;x-amz-content-sha256;x-amz-date
2686acdcb1fdcf33a602537d0383bc85a4ccbca860cbef71825e616c5ddc753d
2021-02-23 15:38:14,473 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210223T153814Z
20210223/us-east-1/s3/aws4_request
25bda79e6b19ac0b289ec1c92bdd8e8b09ebe8267651ea36587baae789ee05eb
2021-02-23 15:38:14,473 - botocore.auth - DEBUG - Signature: ac89fd2880b54fa23a2a1bee7b0e4ca5ecf16dabc9c3b97d5241a1f3a9667944
2021-02-23 15:38:14,473 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://example-test-bucket.s3.amazonaws.com/?delete, headers={'User-Agent': b'Boto3/1.17.13 Python/3.6.8 Linux/3.10.0-1160.15.2.el7.x86_64 Botocore/1.20.13', 'X-Amz-Date': b'20210223T153814Z', 'X-Amz-Content-SHA256': b'2686acdcb1fdcf33a602537d0383bc85a4ccbca860cbef71825e616c5ddc753d', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAWVKXYV4EDOFJRFND/20210223/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ac89fd2880b54fa23a2a1bee7b0e4ca5ecf16dabc9c3b97d5241a1f3a9667944', 'Content-Length': '179'}>
2021-02-23 15:38:14,473 - botocore.httpsession - DEBUG - Certificate path: /home/example/.local/lib/python3.6/site-packages/botocore/cacert.pem
2021-02-23 15:38:14,477 - urllib3.connectionpool - DEBUG - https://example-test-bucket.s3.amazonaws.com:443 "POST /?delete HTTP/1.1" 400 None
2021-02-23 15:38:14,477 - botocore.parsers - DEBUG - Response headers: {'x-amz-request-id': '951AF3AD1934169F', 'x-amz-id-2': 'HfysU3wb1OsJ8p7fJSzJlxKakuJgugtXH4yLB+d55NzyqJPQru0NIveczIeYEmTHzT2FJ4GHfBo=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Tue, 23 Feb 2021 15:38:14 GMT', 'Server': 'AmazonS3', 'Connection': 'close'}
2021-02-23 15:38:14,478 - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidRequest</Code><Message>Missing required header for this request: Content-MD5</Message><RequestId>951AF3AD1934169F</RequestId><HostId>HfysU3wb1OsJ8p7fJSzJlxKakuJgugtXH4yLB+d55NzyqJPQru0NIveczIeYEmTHzT2FJ4GHfBo=</HostId></Error>'
2021-02-23 15:38:14,480 - botocore.hooks - DEBUG - Event needs-retry.s3.DeleteObjects: calling handler <botocore.retryhandler.RetryHandler object at 0x7f39d263ea20>
2021-02-23 15:38:14,480 - botocore.retryhandler - DEBUG - No retry needed.
2021-02-23 15:38:14,480 - botocore.hooks - DEBUG - Event needs-retry.s3.DeleteObjects: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f39d263e9e8>>

[rickatnight11]

Ahah, just determined that this is only happening on an enterprise linux system with FIPS mode enabled, which of course disables MD5. Boto needs to handle this case gracefully.

[zdutta]

Thank you for your post @rickatnight11. Marking this as a feature request and taking a look.

[mliudev]

Having the same issue since we run on FIPS enabled machines: Looks like this is related: https://github.com/boto/botocore/issues/1700

[BongoEADGC6]

@zdutta Any update surrounding this issue?