botocore icon indicating copy to clipboard operation
botocore copied to clipboard
verified publisher icon boto

Performance regression in S3 URL signing

Open akx opened this issue 5 months ago • 4 comments

Describe the bug

While profiling our app, I noticed quite a bit of time being spent signing URLs with Boto3/Botocore, so I decided to dig a little deeper, and turns out that there's been pretty significant performance regressions (that aren't helped by awscrt) – namely, current versions of botocore are around 3x slower when calling S3.Client.get_presigned_url.

The below results were generated with this benchmark code on my macOS machine.

boto3 botocore awscrt count mean (ms) median (ms) std (ms) min (ms) max (ms) Visual Chart
1.20.54 1.23.54 - 10 680.1 676.1 20.3 658.0 720.9 ███████░░░░░░░░░░░░░
1.20.54 1.23.54 0.27.4 10 600.8 592.3 19.3 586.5 648.4 ██████░░░░░░░░░░░░░░
1.21.46 1.24.46 - 10 671.7 666.3 21.6 654.9 729.1 ███████░░░░░░░░░░░░░
1.21.46 1.24.46 0.27.4 10 590.9 589.0 8.3 582.1 611.0 ██████░░░░░░░░░░░░░░
1.22.13 1.25.13 - 10 657.2 656.1 10.0 646.1 678.2 ███████░░░░░░░░░░░░░
1.22.13 1.25.13 0.27.4 10 583.9 582.3 8.8 569.8 600.7 ██████░░░░░░░░░░░░░░
1.23.10 1.26.10 - 10 649.6 650.0 4.0 640.8 654.0 ███████░░░░░░░░░░░░░
1.23.10 1.26.10 0.27.4 10 588.5 587.0 11.5 571.8 614.7 ██████░░░░░░░░░░░░░░
1.24.96 1.27.96 - 10 658.7 651.8 27.3 643.8 735.5 ███████░░░░░░░░░░░░░
1.24.96 1.27.96 0.27.4 10 593.6 591.6 12.2 582.3 621.1 ██████░░░░░░░░░░░░░░
1.25.5 1.28.5 - 10 867.0 868.2 5.7 855.3 876.4 █████████░░░░░░░░░░░
1.25.5 1.28.5 0.27.4 10 791.7 790.6 4.3 786.1 798.0 █████████░░░░░░░░░░░
1.26.165 1.29.165 - 10 866.3 866.8 5.7 858.3 876.4 █████████░░░░░░░░░░░
1.26.165 1.29.165 0.27.4 10 799.9 798.7 10.6 788.5 822.9 █████████░░░░░░░░░░░
1.27.1 1.30.1 - 10 868.2 866.5 10.2 853.4 892.5 █████████░░░░░░░░░░░
1.27.1 1.30.1 0.27.4 10 813.1 804.6 26.9 796.0 885.0 █████████░░░░░░░░░░░
1.28.85 1.31.85 - 10 862.5 860.0 8.9 851.1 876.2 █████████░░░░░░░░░░░
1.28.85 1.31.85 0.27.4 10 803.9 803.9 10.3 789.3 822.7 █████████░░░░░░░░░░░
1.29.7 1.32.7 - 10 1567.2 1562.6 31.6 1534.6 1636.7 █████████████████░░░
1.29.7 1.32.7 0.27.4 10 1747.7 1696.8 173.2 1578.1 2121.9 ████████████████████
1.33.13 1.33.13 - 10 1615.0 1606.5 32.2 1589.0 1697.3 ██████████████████░░
1.33.13 1.33.13 0.27.4 10 1553.3 1547.6 24.4 1521.6 1607.0 █████████████████░░░
1.34.162 1.34.162 - 10 1622.3 1620.6 14.7 1594.0 1650.8 ██████████████████░░
1.34.162 1.34.162 0.27.4 10 1562.7 1564.1 21.5 1534.2 1596.0 █████████████████░░░
1.35.99 1.35.99 - 10 1632.7 1632.2 11.5 1615.9 1652.2 ██████████████████░░
1.35.99 1.35.99 0.27.4 10 1588.0 1577.0 30.7 1564.6 1652.2 ██████████████████░░
1.36.26 1.36.26 - 10 1648.2 1645.8 26.3 1617.4 1706.7 ██████████████████░░
1.36.26 1.36.26 0.27.4 10 1586.0 1578.0 19.0 1569.6 1624.0 ██████████████████░░
1.37.38 1.37.38 - 10 1692.2 1688.0 16.5 1676.7 1725.5 ███████████████████░
1.37.38 1.37.38 0.27.4 10 1591.2 1589.9 12.8 1571.4 1611.4 ██████████████████░░
1.38.46 1.38.46 - 10 1673.3 1667.4 17.4 1650.5 1703.1 ███████████████████░
1.38.46 1.38.46 0.27.4 10 1646.0 1624.9 65.5 1602.4 1824.1 ██████████████████░░

Of course it's not generally feasible to run old Botocore versions in production.

Regression Issue

  • [x] Select this option if this issue appears to be a regression.

Expected Behavior

URL signing to be as fast as it has been with older versions of Botocore.

For the time being, I wrote a small alternative implementation of S3V4 signing that runs circles around the Botocore code.

Current Behavior

Please see above.

Reproduction Steps

Via https://github.com/akx/botocore-sign-perf:

import boto3

s3_client = boto3.client(
    "s3",
    aws_access_key_id="AKIAI" * 10,
    aws_secret_access_key="nom" * 10,
    region_name="eu-west-1",
    config=Config(
        signature_version="s3v4",
        s3={"addressing_style": "path"},
    ),
)

t0 = time.time()
for x in range(10_000):
    s3_client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": "bucket", "Key": str(x)},
        ExpiresIn=120,
    )
t1 = time.time()
sign_time_ms = (t1 - t0) * 1000

Possible Solution

Based on comparing the pyinstrument reports

uv run --with=pyinstrument --with=botocore==1.23.54 --with=boto3 pyinstrument --show='*/boto*' -o old.html main.py
uv run --with=pyinstrument --with=botocore==1.38.46 --with=boto3 pyinstrument --show='*/boto*' -o new.html main.py

it looks like the new versions spend more time in S3._resolve_endpoint_ruleset (that doesn't exist in 1.23.54) than in RequestSigner itself.

I suspect that has to do with evaluating the DSL in the 10k+ line JSON file https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json 😄

Maybe someone should take a critical look at whether EndpointRulesetResolver could cache results, or if it can be sidestepped altogether.

Additional Information/Context

No response

SDK version used

Multiple, see above.

Environment details (OS name and version, etc.)

macOS, but this is reproducible anywhere.

akx avatar Jul 22 '25 15:07 akx

Thanks to @hannesj for digging around the JS side of the fence – the AWS JS SDK has a cache for endpoint resolution

Botocore also attempts to cache these: https://github.com/boto/botocore/blob/7ddc232fff4297fc19f89a70ce95014beb2d77bf/botocore/endpoint_provider.py#L707-L708

However, the effectiveness of that cache is near-zero unless you're signing the exact same key over and over, as Key is one of the parameters being passed to it via https://github.com/boto/botocore/blob/7ddc232fff4297fc19f89a70ce95014beb2d77bf/botocore/regions.py#L502-L509 (as it is defined in https://github.com/boto/botocore/blob/7ddc232fff4297fc19f89a70ce95014beb2d77bf/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json#L61-L65).

However, no rules in endpoint-rule-set-1.json actually refer to Key, so it probably shouldn't be defined as a endpoint ruleset parameter. It looks like it would be worth it to figure out why it's ended up there (and/or if there are other extraneous keys) – the bot commit 5d97d5b7e47705bdb66356ec3616566b18e2d00c introducing it doesn't help much 😉

I can confirm that just removing the Key param from that JSON file has a delightful effect already:

$ uv run main.py
{"sign_time_ms": 1649.1360664367676, "python_version": [3, 12, 11, "final", 0], "boto3_version": "1.39.10", "botocore_version": "1.39.10", "awscrt_version": null, "hyperfine_iteration": null}
$ uv pip install -e ../botocore
$ uv run main.py
{"sign_time_ms": 914.9761199951172, "python_version": [3, 12, 11, "final", 0], "boto3_version": "1.39.10", "botocore_version": "1.39.10-devel", "awscrt_version": null, "hyperfine_iteration": null}
$

akx avatar Jul 22 '25 17:07 akx

Hello @akx, thanks for reaching out. The team is aware and looking at this issue. For any updates, we will reply here but for now this issue will remain open for tracking.

adev-code avatar Jul 24 '25 16:07 adev-code

@adev-code Thanks. Please see the PR linked above.

akx avatar Jul 24 '25 16:07 akx

@adev-code Any movement on this?

Newer versions of Botocore don't seem to be helping.

boto3 botocore awscrt count mean (ms) median (ms) std (ms) min (ms) max (ms) Visual Chart
1.39.10 1.39.10 0.28.4 30 1625.1 1622.7 35.1 1570.0 1718.0 ██████████████████░░
1.39.10 1.40.71 - 10 1691.8 1671.7 59.3 1614.7 1769.7 ███████████████████░
1.39.10 1.40.71 0.28.4 10 1571.8 1558.2 42.5 1535.8 1662.5 █████████████████░░░

akx avatar Nov 12 '25 07:11 akx