upgrade from 2.2.X to 2.3.X breaks tests using botocore.stub.Stubber

[akursar]

Describe the bug Upgrading from 2.2.X to 2.3.X causes tests that use Stubber to fail. I have this example test, which is an aiobotocore version of the example described at https://botocore.amazonaws.com/v1/documentation/api/latest/reference/stubber.html#botocore.stub.Stubber

The test passes if I downgrade to 2.2.0, but fails with 2.3.0 (and 2.3.2).

I'm not sure if aiobotocore seeks to be compatible with Stubber, but it seemed worthwhile to report the issue.

import datetime

import pytest
from aiobotocore.session import get_session
from botocore.stub import Stubber


@pytest.mark.asyncio
async def test_example():
    session = get_session()
    async with session.create_client('s3') as s3:
        stubber = Stubber(s3)

        response = {
            'IsTruncated': False,
            'Name': 'test-bucket',
            'MaxKeys': 1000, 'Prefix': '',
            'Contents': [{
                'Key': 'test.txt',
                'ETag': '"abc123"',
                'StorageClass': 'STANDARD',
                'LastModified': datetime.datetime(2016, 1, 20, 22, 9),
                'Owner': {'ID': 'abc123', 'DisplayName': 'myname'},
                'Size': 14814
            }],
            'EncodingType': 'url',
            'ResponseMetadata': {
                'RequestId': 'abc123',
                'HTTPStatusCode': 200,
                'HostId': 'abc123'
            },
            'Marker': ''
        }

        expected_params = {'Bucket': 'test-bucket'}

        stubber.add_response('list_objects', response, expected_params)
        stubber.activate()

        service_response = await s3.list_objects(Bucket='test-bucket')
        assert service_response == response

It passes for 2.2.0 but fails for 2.3.0 with

aiohttp.client_exceptions.ServerTimeoutError: Connection timeout to host http://169.254.169.254/latest/api/token

Checklist

• [X] I have reproduced in environment where pip check passes without errors
• [X] I have provided pip freeze results
• [X] I have provided sample code or detailed way to reproduce
• [X] I have tried the same code in botocore to ensure this is an aiobotocore specific issue
• [X] I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue

sort of. i am not changing the version of aiohttp but am only updating aiobotocore

• [X] I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore==2.3.2
aiohttp==3.8.1
aioitertools==0.10.0
aiosignal==1.2.0
async-timeout==4.0.2
attrs==21.4.0
botocore==1.24.21
charset-normalizer==2.0.12
frozenlist==1.3.0
idna==3.3
iniconfig==1.1.1
jmespath==1.0.0
multidict==6.0.2
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.9
pytest==7.1.2
pytest-asyncio==0.18.3
python-dateutil==2.8.2
six==1.16.0
tomli==2.0.1
typing_extensions==4.2.0
urllib3==1.26.9
wrapt==1.14.1
yarl==1.7.2

Environment:

> python --version
Python 3.8.13
> uname -ma
Darwin NY-AKursar 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:45:05 PDT 2022; root:xnu-8020.101.4~15/RELEASE_X86_64 x86_64
> 

[takeda]

I also run in the same issue, version 2.3.0 (required by aioboto3 9.6.0) on python 3.10.4.

Unit tests that worked fine in 2.3.0 and older no longer work.

[thehesiod]

will look asap, sorry for delay

[takeda]

@thehesiod any updates?

[takeda]

@thehesiod I see updates being made, but this is still a problem. I am forced to fix to version 2.2.0 and can't benefit of any improvements being made. Is really no one using botocore's built-in Stubber for unit tests?

[takeda]

@thehesiod @jakob-keller is there a chance the Stubber is ever going to be fixed?

Something broke in version 2.3.0 and in other to keep my unit tests I have to keep aioboto3 at 9.5.0 and aiobotocore at 2.2.0 and right now that's awfully old version :(

[takeda]

This is the PR that introduced the problem: https://github.com/aio-libs/aiobotocore/pull/934

[thehesiod]

sorry haven't had a chance to look into this yet

[takeda]

@thehesiod any idea what could be broken? I could try to fix it, but I don't even know where to start. I isolated the PR via bisect.

[thehesiod]

i tried that test above in the latest aiobotocore and it passes after adding @pytest.mark.moto

[takeda]

The @pytest.mark.moto applies moto package which monkeypatches boto3 to emulate AWS services.

Stubber is builtin component of botocore that is meant for unit testing: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/stubber.html

After 2.3.0 change the Stubber() object is a NO-OP, the API still tries to talk to AWS.

[thehesiod]

ok will re-try, it's because right now our test infra doesn't run tests that don't have moto enabled, I'll try it w/o

[takeda]

any luck?

[takeda]

So I started migrating to moto, which apparently is broken and currently also need workaround and also doesn't let me test everything. So since either doesn't work correctly I started working with simple example so I used one in this bug. Interestingly this unit tests passes in latest version. I tried to do bisect but git complained that working version and broken don't share common ancestor (is the code rebased on every major version?)

Anyway, I traced it manually to this PR: https://github.com/aio-libs/aiobotocore/pull/989 it baffles me how this change somehow fixed the issue for this access.

I still do have problem though, with ec2, ssm and elbv2 so I will try to come up with another unit test that fails.

[takeda]

So after more investigation the issue was also introduced by https://github.com/aio-libs/aiobotocore/pull/934 as well.

Basically I created a fixture to use Stubber, like this:

@pytest_asyncio.fixture()
async def fake_ssm():
    async with aioboto3.Session().client("ssm") as client:
        fake_session = create_autospec(aioboto3.Session, spec_set=True)
        fake_session.client = MagicMock(return_value=client)

        ssm_obj = ssm.SSM(b3_session=fake_session)
        stubber = Stubber(client)

        with stubber:
            yield FakeSSM(ssm_obj, stubber)

        stubber.assert_no_pending_responses()

this now produces assertion error:

/nix/store/3zg1r17c2nq6d7wk5gnclsk96458qrcs-python3-3.11.4-env/lib/python3.11/site-packages/aiobotocore/client.py:600: in __aenter__
    await self._endpoint.http_session.__aenter__()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <aiobotocore.httpsession.AIOHTTPSession object at 0x1111c1310>

    async def __aenter__(self):
>       assert not self._session and not self._connector
E       AssertionError

/nix/store/3zg1r17c2nq6d7wk5gnclsk96458qrcs-python3-3.11.4-env/lib/python3.11/site-packages/aiobotocore/httpsession.py:120: AssertionError

This worked in 2.2.3 It looks like now if client is openned, it can't be opened again. I modified it to do this:

@pytest_asyncio.fixture()
async def fake_ssm():
    async with aioboto3.Session().client("ssm") as client:
        fake_session = create_autospec(aioboto3.Session, spec_set=True)
        fake_session.client = MagicMock(return_value=client)

        ssm_obj = ssm.SSM(b3_session=fake_session)
        stubber = Stubber(client)

    with stubber:
        yield FakeSSM(ssm_obj, stubber)

    stubber.assert_no_pending_responses()

and it seems to be working again.

As far as I'm concerned this issue is resolved. @akursar, is it also working for you?

[thehesiod]

so I tested the original testcase w/o moto and it passes fine with latest version, shall we close?

[takeda]

I think so. I needed to adapt my code a bit, but looking at the assertion I think I was maybe using it incorrectly and the old version was just allowing it.

[thehesiod]

btw using the sync stubber is not really correct, as it doesn't return the AioAWSResponse...technically we should have an AioStubber...however unless you need the functionality no biggie

[takeda]

I don't know internals of botocore, but I got impression that stubber just adds hooks to the client and provides responses set up with add_response() whit are just dictionaries.

It seems to work so far, but if there might be some issues, I'm interested what would need to be changed?

[thehesiod]

stuff like https://github.com/boto/botocore/blob/develop/botocore/stub.py#L247 should return https://github.com/aio-libs/aiobotocore/blob/88a43098b1194bf759581acc710ad5bdbdd99a96/aiobotocore/awsrequest.py#L5 to mimic aiobotocore responses

[takeda]

@thehesiod I myself don't use the headers so far, but created PR, is that what were you thinking: https://github.com/aio-libs/aiobotocore/pull/1039