boto3
boto3 copied to clipboard
boto
Announcement: S3 default integrity change
In AWS SDK for Python v1.36.0, we released changes to the S3 client that adopts new default integrity protections. For more information on default integrity behavior, please refer to the official SDK documentation. In SDK releases from this version on, clients default to enabling an additional checksum on all Put calls and enabling validation on Get calls.
You can disable default integrity protections for S3. We do not recommend this because checksums are important to S3 integrity posture. Integrity protections can be disabled by setting the config flag to when_required, or by using the related AWS shared config file settings or environment variables.
Disclaimer: The AWS SDKs and CLI are designed for usage with official AWS services. We may introduce and enable new features by default, such as these new default integrity protections prior to them being supported or handled by third-party service implementations. You can disable the new behavior with the when_required value for the request_checksum_calculation and response_checksum_validation configuration options covered in Data Integrity Protections for Amazon S3.
They did the same thing in the JS client. Leave it to AWS to introduce breaking changes and pretend their library isn't used extensively with other services...
+1
ImportError: cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum'
For those using third-party service implementations, please see the updated disclaimer above:
Disclaimer: The AWS SDKs and CLI are designed for usage with official AWS services. We may introduce and enable new features by default, such as these new default integrity protections prior to them being supported or handled by third-party service implementations. You can disable the new behavior with the
WHEN_REQUIREDvalue for therequest_checksum_calculationandresponse_checksum_validationconfiguration options covered in Data Integrity Protections for Amazon S3.
ImportError: cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum'
@FChmiel this is most likely a result of not using the latest version of botocore where this constant was introduced (botocore-1.36.0). This is the minimum supported version for boto3-1.36.0, however, if you're using the s3transfer package without boto3, we've identified an relevant issue and merged a fix for this last night which will be included in today's daily release for s3transfer-0.11.1.
Tracking issue: https://github.com/boto/s3transfer/issues/324
Just so you know - this completely broke the sns_extended_client. sns_extended_client is an AWS implementation.
hey @jonathan343, We're encountering the same issue with our integration tests running in the cibuildwheel container. I've reviewed the logs, and we consistently install the latest boto3/botocore version during each test run. This suggests that the issue isn't related to using an outdated version of botocore, as you had proposed. For reference, here are the relevant version details from our logs:
cibuildwheel version 2.22.0
Build options:
platform: linux
allow_empty: False
architectures: aarch64, armv7l, i686, ppc64le, s390x, x86_64
build_selector:
build_config: cp39-manylinux_aarch64
skip_config:
requires_python: <3.13,>=3.8
enable: ['cpython-freethreading', 'cpython-prerelease', 'pypy']
Collecting boto3 (from s3torchconnectorclient==1.3.1->s3torchconnectorclient==1.3.1)
Downloading boto3-1.36.0-py3-none-any.whl.metadata (6.6 kB)
...
Collecting botocore<1.37.0,>=1.36.0 (from boto3->s3torchconnectorclient==1.3.1->s3torchconnectorclient==1.3.1)
Downloading botocore-1.36.0-py3-none-any.whl.metadata (5.7 kB)
Thanks @jonathan343.
I also suspect its not us using an outdated botocore version: We only pin the boto3 version and botocore is a transitive dependancy of this only.
@IsaevIlya @FChmiel Are either of you able to provide a reproducible error? Also, can you verify the correct versions in your environment after installing all dependencies using something like python -m pip freeze?
The following shows me installing the latest version of boto3 from PyPI and successfully importing DEFAULT_CHECKSUM_ALGORITHM from botocore.httpchecksum.
$ python -m venv venv
$ . venv/bin/activate
(venv) $ pip install boto3 --no-cache-dir
Collecting boto3
Downloading boto3-1.36.0-py3-none-any.whl.metadata (6.6 kB)
Collecting botocore<1.37.0,>=1.36.0 (from boto3)
Downloading botocore-1.36.0-py3-none-any.whl.metadata (5.7 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3)
Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.12.0,>=0.11.0 (from boto3)
Downloading s3transfer-0.11.0-py3-none-any.whl.metadata (1.7 kB)
Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.37.0,>=1.36.0->boto3)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3!=2.2.0,<3,>=1.25.4 (from botocore<1.37.0,>=1.36.0->boto3)
Downloading urllib3-2.3.0-py3-none-any.whl.metadata (6.5 kB)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.37.0,>=1.36.0->boto3)
Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Downloading boto3-1.36.0-py3-none-any.whl (139 kB)
Downloading botocore-1.36.0-py3-none-any.whl (13.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.3/13.3 MB 111.8 MB/s eta 0:00:00
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading s3transfer-0.11.0-py3-none-any.whl (84 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading urllib3-2.3.0-py3-none-any.whl (128 kB)
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.36.0 botocore-1.36.0 jmespath-1.0.1 python-dateutil-2.9.0.post0 s3transfer-0.11.0 six-1.17.0 urllib3-2.3.0
(venv) $ pip freeze
boto3==1.36.0
botocore==1.36.0
jmespath==1.0.1
python-dateutil==2.9.0.post0
s3transfer==0.11.0
six==1.17.0
urllib3==2.3.0
(venv) $ python
Python 3.13.1 (main, Dec 12 2024, 00:56:51) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from botocore.httpchecksum import DEFAULT_CHECKSUM_ALGORITHM
>>> DEFAULT_CHECKSUM_ALGORITHM
'CRC32'
However, when I install a version of botocore < 1.36.0 in the same environment (this is a dependency conflict) I get the same ImportError you're seeing:
(venv) $ pip install botocore==1.35.99
Collecting botocore==1.35.99
Using cached botocore-1.35.99-py3-none-any.whl.metadata (5.7 kB)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in ./venv/lib/python3.13/site-packages (from botocore==1.35.99) (1.0.1)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./venv/lib/python3.13/site-packages (from botocore==1.35.99) (2.9.0.post0)
Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in ./venv/lib/python3.13/site-packages (from botocore==1.35.99) (2.3.0)
Requirement already satisfied: six>=1.5 in ./venv/lib/python3.13/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.35.99) (1.17.0)
Using cached botocore-1.35.99-py3-none-any.whl (13.3 MB)
Installing collected packages: botocore
Attempting uninstall: botocore
Found existing installation: botocore 1.36.0
Uninstalling botocore-1.36.0:
Successfully uninstalled botocore-1.36.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.36.0 requires botocore<1.37.0,>=1.36.0, but you have botocore 1.35.99 which is incompatible.
Successfully installed botocore-1.35.99
(venv) $ pip freeze
boto3==1.36.0
botocore==1.35.99
jmespath==1.0.1
python-dateutil==2.9.0.post0
s3transfer==0.11.0
six==1.17.0
urllib3==2.3.0
(venv) $ python
Python 3.13.1 (main, Dec 12 2024, 00:56:51) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from botocore.httpchecksum import DEFAULT_CHECKSUM_ALGORITHM
Traceback (most recent call last):
File "<python-input-0>", line 1, in <module>
from botocore.httpchecksum import DEFAULT_CHECKSUM_ALGORITHM
ImportError: cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum' (/Users/gytndd/Desktop/oncall/venv/lib/python3.13/site-packages/botocore/httpchecksum.py)
Thanks, I replicate your (correct) behaviour with poetry when trying to generate a a reproducible error.
In my lock file I have s3transfer==0.11.0, so that may be it.
@talsalmona - thanks for the report. I've sent the concern to the maintainers of that package. If you can, it would be helpful to provide more details on the failure scenario on the amazon-sns-python-extended-client-lib repository:
https://github.com/awslabs/amazon-sns-python-extended-client-lib/issues
@talsalmona - thanks for the report. I've sent the concern to the maintainers of that package. If you can, it would be helpful to provide more details on the failure scenario on the
amazon-sns-python-extended-client-librepository:https://github.com/awslabs/amazon-sns-python-extended-client-lib/issues
Sure, added here: https://github.com/awslabs/amazon-sns-python-extended-client-lib/issues/20
Hey @jonathan343, I'm unable to reproduce the issue anymore. Today our integration tests are picking new versions with which everything working fine. I suppose whatever issue was, it was resolved for us:
boto3==1.36.1
botocore==1.36.1
...
s3transfer==0.11.1
I saw the s3transfer DEFAULT_CHECKSUM_ALGORITHM issue from a python service that has been running for a few months.
These versions when I arrived at the system:
boto3 1.36.6
botocore 1.36.6
s3transfer 0.11.1
Simply restarting the service resolved the issue.
It could be vaguely possible that the service had not performed the task using s3transfer until after s3transfer was updated so the loaded versions of boto* vs s3transfer were from very different eras. Anyone have a strategy for forcing Python to load all packages during startup?
Our application encountered the following issue (a simple restart did not help, as the issue kept reoccurring):
botocore.exceptions.ClientError: An error occurred (XAmzContentSHA256Mismatch) when calling the PutObject operation: The Content-SHA256 you specified did not match what we received
This occurred while using the following library versions at the time of installation:
boto3==1.36.9
botocore==1.36.9
s3transfer==0.11.2
Simply pinning the versions to the previous minor version resolved our issue:
boto3<1.36.0
botocore<1.36.0
s3transfer<0.11.0
We're currently investigating whether the issue is related to the s3transfer version or solely due to boto3 and botocore. Any insights would be appreciated.
We're currently investigating whether the issue is related to the s3transfer version or solely due to boto3 and botocore. Any insights would be appreciated.
Your s3transfer version is up-to-date. It might be worthwhile to update boto3 and botocore to the most recent releases though, there were some fixes made in recent versions. The only other reports I've seen with this error are using third-party S3-like services. Are you using AWS S3?
Your s3transfer version is up-to-date. It might be worthwhile to update boto3 and botocore to the most recent releases though, there were some fixes made in recent versions.
Alright, thanks for pointing out about the new patch versions.
The only other reports I've seen with this error are using third-party S3-like services. Are you using AWS S3?
No, we are using a third-party S3-compatible service. However, with previous minor versions of boto3 and botocore everything has been working flawlessly.
No, we are using a third-party S3-compatible service. However, with previous minor versions of boto3 and botocore everything has been working flawlessly.
As mentioned in the announcement, if you're using a third-party S3-like service, you might experience issues. You can disable the new behavior with the when_required value for the request_checksum_calculation and response_checksum_validation configuration options. Let me know if you continue to have problems even with those values set.
@RyanFitzSimmonsAK
Hello, I have tried your proposed solution and it is not working on my side.
Can you provide a working sample code so I can reproduce on my side?
My code:
import boto3
from botocore.config import Config
# My existing Linode S3-like configuration
linode_obj_config = {
"aws_access_key_id": "ACCESS_KEY",
"aws_secret_access_key": "SECRET_KEY",
"endpoint_url": "https://gb-lon-1.linodeobjects.com",
}
# Configure to disable checksum calculation and validation
custom_s3_config = Config(
s3={
'request_checksum_calculation': 'when_required',
'response_checksum_validation': 'when_required'
}
)
# Create the client with the custom configuration
client = boto3.client("s3", config=custom_s3_config, **linode_obj_config)
# List buckets
response = client.list_buckets()
for bucket in response['Buckets']:
print(bucket['Name'])
# Upload a file
client.upload_file(Filename='test.png', Bucket='whatsapp-profiles', Key='test.png')
Thanks in advance.
Edit: Added my code, formatting issue
Hello, I have tried your proposed solution and it is not working on my side.
Can you provide a working sample code so I can reproduce on my side?
request_checksum_calculation and response_checksum_validation shouldn't be under the S3-specific configuration options.
custom_s3_config = Config(
request_checksum_calculation = 'when_required',
response_checksum_validation = 'when_required',
)
Hello, I have tried your proposed solution and it is not working on my side. Can you provide a working sample code so I can reproduce on my side?
request_checksum_calculationandresponse_checksum_validationshouldn't be under the S3-specific configuration options.custom_s3_config = Config( request_checksum_calculation = 'when_required', response_checksum_validation = 'when_required', )
Worked. Thanks!
After following the suggestion under this announcement (setting request_checksum_calculation = 'when_required',
response_checksum_validation = 'when_required',) in s3fs I get
the Delete operation fails with botocore.errorfactory.InvalidRequest: An error occurred (InvalidRequest) when calling the DeleteObjects operation: Missing required header for this request: Content-MD5
Why did you screw up your library? People don't use solely AWS S3.
After following the suggestion under this announcement (setting request_checksum_calculation = 'when_required', response_checksum_validation = 'when_required',) in s3fs I get the Delete operation fails with botocore.errorfactory.InvalidRequest: An error occurred (InvalidRequest) when calling the DeleteObjects operation: Missing required header for this request: Content-MD5
DeleteObjects is one of the operations that do require a checksum, so setting request_checksum_calculation to when_required wouldn't disable it. Operations in the S3 model file (https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/service-2.json) with "requestChecksumRequired":true have this behavior. Unfortunately, there isn't currently a good workaround beyond noting that DeleteObject (as opposed to DeleteObjects) does not require a checksum.
FYI to anyone using awscli, you want env-vars AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION both set to when_required: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html#envvars-list-AWS_REQUEST_CHECKSUM_CALCULATION
