Integrity check on put_object not working as expected

[junqfisica]

Describe the bug

The kwargs: ChecksumCRC32, ChecksumCRC32C, ChecksumSHA1, and ChecksumSHA256, for client.put_object() have not effect on checking data integrity. Only ContentMD5 works.

I'm not sure if this is a documentation problem or an actual bug.

Expected Behavior

I would expect this to fail:

response = client.put_object( Body='filetoupload', Bucket='examplebucket', Key='objectkey', ChecksumSHA1='wrong_hash' )

Since I'm passing a wrong hash for the ChecksumSHA1.

Current Behavior

This method successfully uploads my file.

response = client.put_object( Body='filetoupload', Bucket='examplebucket', Key='objectkey', ChecksumSHA1='wrong_hash' )

Reproduction Steps

1 - Start your client:

import boto3

client = boto3.client(
    service_name='s3', 
    endpoint_url=S3_ENDPOINT, 
    aws_access_key_id=S3_ACCESS_KEY_ID,  
    aws_secret_access_key=S3_SECRET_ACCESS_KEY,
)

2 - Upload an object to s3 using a wrong checksum for ChecksumSHA1:

response = client.put_object(
    Body='filetoupload',
    Bucket='examplebucket',
    Key='objectkey',
    ChecksumSHA1='wrong_hash'
)

print(response)

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.21.43

Environment details (OS name and version, etc.)

Windows 10 Python 3.8

[kdaily]

Hi @junqfisica,

Thanks for writing in. I'm not able to reproduce, using boto3==1.21.43. Here's what I get:

In [2]: import boto3
   ...: client = boto3.client('s3')

In [3]: response = client.put_object(
   ...:     Body='test.txt',
   ...:     Bucket='mybucket',
   ...:     Key='mykey/test.txt',
   ...:     ChecksumSHA1='wrong_hash'
   ...: )
   ...:
   ...:
---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 response = client.put_object(
      2     Body='test.txt',
      3     Bucket='mybucket',
      4     Key='mykey/test.txt',
      5     ChecksumSHA1='wrong_hash'
      6 )

File ~/.pyenv/versions/3.8.8/envs/boto3-latest-py3.8.8/lib/python3.8/site-packages/botocore/client.py:415, in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
    412     raise TypeError(
    413         "%s() only accepts keyword arguments." % py_operation_name)
    414 # The "self" in this scope is referring to the BaseClient.
--> 415 return self._make_api_call(operation_name, kwargs)

File ~/.pyenv/versions/3.8.8/envs/boto3-latest-py3.8.8/lib/python3.8/site-packages/botocore/client.py:745, in BaseClient._make_api_call(self, operation_name, api_params)
    743     error_code = parsed_response.get("Error", {}).get("Code")
    744     error_class = self.exceptions.from_code(error_code)
--> 745     raise error_class(parsed_response, operation_name)
    746 else:
    747     return parsed_response

ClientError: An error occurred (InvalidRequest) when calling the PutObject operation: Value for x-amz-checksum-sha1 header is invalid.

Doing this in with debugging on, I can see the 400 response from the server:

2022-04-21 16:36:47,962 urllib3.connectionpool [DEBUG] https://mybucket.s3.us-west-2.amazonaws.com:443 "PUT /mykey/test.txt HTTP/1.1" 400 None
2022-04-21 16:36:47,963 botocore.parsers [DEBUG] Response headers: {'x-amz-request-id': 'HRGMGHRCFPNN4YHG', 'x-amz-id-2': 'G7lIMb/MRY642gdpwKF5xxbr1h6lGVqxT0IHyAAt7CtwMiokjH22tNic3PQQbXkftdbVvTWnUdA=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Thu, 21 Apr 2022 23:36:47 GMT', 'Server': 'AmazonS3', 'Connection': 'close'}
2022-04-21 16:36:47,963 botocore.parsers [DEBUG] Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidRequest</Code><Message>Value for x-amz-checksum-sha1 header is invalid.</Message><RequestId>HRGMGHRCFPNN4YHG</RequestId><HostId>G7lIMb/MRY642gdpwKF5xxbr1h6lGVqxT0IHyAAt7CtwMiokjH22tNic3PQQbXkftdbVvTWnUdA=</HostId></Error>'

Can you provide debug logs using boto3.set_stream_logger()? Please redact any sensitive details.

A few specific questions too:

1. Which AWS region are you attempting this in?
2. What is your botocore version?

[junqfisica]

Hi @kdaily,

Thank you for your reply. That's interesting I'm still not getting any errors when putting the object with a wrong hash.

About your two questions:

1. The debug says my region is us-east-1. However, we are using self-host S3 object storage, The service is provided by the state of Brandenburg in Germany.
2. botocore = 1.24.43

Running this:

response = client.put_object(Body='test.txt', Bucket=s3m.bucket_name, Key='mykey/test.txt', ChecksumSHA1='wrong_hash')
print(response)

Gives back:

{'ResponseMetadata': {'RequestId': 'd1cfae75e549995699d7', 'HostId': 'd1cfae75e549995699d7', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'nginx', 'date': 'Fri, 22 Apr 2022 09:10:43 GMT', 'transfer-encoding': 'chunked', 'etag': '"dd18bf3a8e0a2a3e53e2661c7fb53534"', 'x-amz-id-2': 'd1cfae75e549995699d7', 'x-amz-request-id': 'd1cfae75e549995699d7'}, 'RetryAttempts': 0}, 'ETag': '"dd18bf3a8e0a2a3e53e2661c7fb53534"'}

As you can see the response doesn't include ChecksumSHA1

Here is my debug log:

2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x0000023872664AF0>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x000002387265BEE0>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x0000023872665430>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x000002387265BE50>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x00000238766FBD60>>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x00000238766FBE20>>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x000002387265BCA0>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-call.s3.PutObject: calling handler <function conditionally_calculate_md5 at 0x000002387250E040>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x00000238726641F0>
2022-04-22 11:10:43,783 botocore.handlers [DEBUG] Adding expect 100 continue header to request.
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-call.s3.PutObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x00000238766FBD60>>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-call.s3.PutObject: calling handler <function add_recursion_detection_header at 0x000002387265B940>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-call.s3.PutObject: calling handler <function inject_api_version_header_if_needed at 0x0000023872665550>
2022-04-22 11:10:43,783 botocore.endpoint [DEBUG] Making request for OperationModel(name=PutObject) with params: {'url_path': '/digiproduction-development/mykey/test.txt', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-checksum-sha1': 'wrong_hash', 'User-Agent': 'Boto3/1.21.43 Python/3.8.5 Windows/10 Botocore/1.24.43', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x00000238767CA680>, 'url': 'http://10.x.x.x/digiproduction-development/mykey/test.txt', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x00000238766FB250>, 'has_streaming_input': True, 'auth_type': None, 'signing': {'bucket': 'digiproduction-development'}}}
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x00000238766FB040>>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x00000238766FBEB0>>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x00000238762E4070>>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x000002387265BB80>
2022-04-22 11:10:43,783 botocore.hooks [DEBUG] Event before-sign.s3.PutObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x00000238766FBEB0>>
2022-04-22 11:10:43,783 botocore.utils [DEBUG] Using S3 path style addressing.
2022-04-22 11:10:43,783 botocore.auth [DEBUG] Calculating signature using v4 auth.
2022-04-22 11:10:43,783 botocore.auth [DEBUG] CanonicalRequest:
PUT
/digiproduction-development/mykey/test.txt

host:10.x.x.x
x-amz-checksum-sha1:wrong_hash
x-amz-content-sha256:a6ed0c785d4590bc95c216bcf514384eee6765b1c2b732d0b0a1ad7e14d3204a
x-amz-date:20220422T091043Z

host;x-amz-checksum-sha1;x-amz-content-sha256;x-amz-date
a6ed0c785d4590bc95c216bcf514384eee6765b1c2b732d0b0a1ad7e14d3204a
2022-04-22 11:10:43,799 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20220422T091043Z
20220422/us-east-1/s3/aws4_request
2f5a71444d6f37b6298052da50053cba2a23ea752b349368e12dc21267d04353
2022-04-22 11:10:43,799 botocore.auth [DEBUG] Signature:
511b16cb0ddb1d158121a4ec7ea7841cf526bc47279e4f44dbed1832b477d67c
2022-04-22 11:10:43,799 botocore.hooks [DEBUG] Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x0000023872665C10>
2022-04-22 11:10:43,799 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://10.x.x.x/digiproduction-development/mykey/test.txt, headers={'x-amz-checksum-sha1': b'wrong_hash', 'User-Agent': b'Boto3/1.21.43 Python/3.8.5 Windows/10 Botocore/1.24.43', 'Expect': b'100-continue', 'X-Amz-Date': b'20220422T091043Z', 'X-Amz-Content-SHA256': b'a6ed0c785d4590bc95c216bcf514384eee6765b1c2b732d0b0a1ad7e14d3204a', 'Authorization': b'AWS4-HMAC-SHA256 Credential=x/x/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-checksum-sha1;x-amz-content-sha256;x-amz-date, Signature=511b16cb0ddb1d158121a4ec7ea7841cf526bc47279e4f44dbed1832b477d67c', 'amz-sdk-invocation-id': b'c7b016ae-8e42-4b50-9c6e-819cf9df5ea5', 'amz-sdk-request': b'attempt=1', 'Content-Length': '8'}>
2022-04-22 11:10:43,799 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 10.x.x.x:80
2022-04-22 11:10:43,799 botocore.awsrequest [DEBUG] Waiting for 100 Continue response.
2022-04-22 11:10:43,799 botocore.awsrequest [DEBUG] 100 Continue response seen, now sending request body.
2022-04-22 11:10:43,830 urllib3.connectionpool [DEBUG] http://10.x.x.x:80 "PUT /digiproduction-development/mykey/test.txt HTTP/1.1" 200 None
2022-04-22 11:10:43,830 botocore.parsers [DEBUG] Response headers: {'Server': 'nginx', 'Date': 'Fri, 22 Apr 2022 09:10:43 GMT', 'Transfer-Encoding': 'chunked', 'ETag': '"dd18bf3a8e0a2a3e53e2661c7fb53534"', 'x-amz-id-2': 'd1cfae75e549995699d7', 'x-amz-request-id': 'd1cfae75e549995699d7'}
2022-04-22 11:10:43,830 botocore.parsers [DEBUG] Response body:
b''
2022-04-22 11:10:43,830 botocore.hooks [DEBUG] Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x00000238766FBD00>
2022-04-22 11:10:43,830 botocore.retryhandler [DEBUG] No retry needed.
2022-04-22 11:10:43,830 botocore.hooks [DEBUG] Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x00000238766FBD60>>

Ps: I just edited the IP address for security reasons.

[junqfisica]

Update:

After debugging this problem a bit further. It seems my response is not returning the header x-amz-checksum-sha1

I put a breakpoint at line 439 and at line 451

I can verify that request.headers are:

request.headers = {'x-amz-checksum-sha1': b'wrong_hash', 'User-Agent': b'Boto3/1.21.43 Python/3.8.5 Windows/10 Botocore/1.24.43', 'Expect': b'100-continue', 'X-Amz-Date': b'20220422T123736Z', 'X-Amz-Content-SHA256': b'a6ed0c785d4590bc95c216bcf514384eee6765b1c2b732d0b0a1ad7e14d3204a', 'Authorization': x/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-checksum-sha1;x-amz-content-sha256;x-amz-date, Signature='x', 'amz-sdk-invocation-id': b'bb6da3c6-6307-4698-905d-2afb98a27552', 'amz-sdk-request': b'attempt=1', 'Content-Length': '8'}

However, my response headers are:

urllib_response.headers = HTTPHeaderDict({'Server': 'nginx', 'Date': 'Fri, 22 Apr 2022 12:38:27 GMT', 'Transfer-Encoding': 'chunked', 'ETag': '"dd18bf3a8e0a2a3e53e2661c7fb53534"', 'x-amz-id-2': 'e72bbdbfdc4b8859a76c', 'x-amz-request-id': 'e72bbdbfdc4b8859a76c'})

I'm using urllib3 = 1.25.10

The question now is, how dependent the checksum kwargs are on the used S3 service (i.e version or extra configuration needed)? Besides that, I have no clue why this is not working on my end.

Thanks in advance

[aBurmeseDev]

Just checking in here to see if I could help. Are you still experiencing the issue? Please let me know if there's any specific question I could help with!

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.