Aiobotocore KeyError: 'retries'

[jowparks]

I am using s3fs with pyarrow, and I am running into a pretty cryptic bug about a key error value. The file begins to write to s3, but then fails with the following exception:

WARNING:aiobotocore.credentials:Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/credentials.py", line 292, in _protected_refresh
    metadata = await self._refresh_using()
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/credentials.py", line 346, in fetch_credentials
    return await self._get_cached_credentials()
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/credentials.py", line 356, in _get_cached_credentials
    response = await self._get_credentials()
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/credentials.py", line 416, in _get_credentials
    return await client.assume_role_with_web_identity(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/client.py", line 173, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/client.py", line 193, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/endpoint.py", line 81, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/retryhandler.py", line 189, in __call__
    retries_context = kwargs['request_dict']['context']['retries']
KeyError: 'retries'

My pip packages are:

aiobotocore==2.1.0
aiohttp==3.8.1
aioitertools==0.8.0
aiosignal==1.2.0
asn1crypto==1.4.0
async-timeout==4.0.2
attrs==21.4.0
awscli==1.22.30
azure-common==1.1.27
azure-core==1.21.1
azure-storage-blob==12.9.0
biopython==1.79
boto3==1.20.30
botocore==1.23.30
certifi==2020.12.5
cffi==1.15.0
chardet==4.0.0
charset-normalizer==2.0.10
colorama==0.4.3
cryptography==3.4.8
docutils==0.15.2
frozenlist==1.3.0
fsspec==2022.1.0
idna==2.10
iniconfig==1.1.1
isodate==0.6.1
jmespath==0.10.0
msrest==0.6.21
multidict==6.0.2
numpy==1.22.1
oauthlib==3.1.1
oscrypto==1.2.1
packaging==21.3
pandas==1.2.5
pluggy==0.13.1
py==1.11.0
pyarrow==3.0.0
pyasn1==0.4.8
pycparser==2.21
pycryptodomex==3.12.0
PyJWT==2.3.0
pyOpenSSL==20.0.1
pyparsing==3.0.6
pysam==0.18.0
pytest==6.2.4
pytest-mock==3.6.1
python-dateutil==2.8.2
pytz==2021.3
PyYAML==5.4.1
requests==2.27.1
requests-oauthlib==1.3.0
rsa==4.7.2
s3fs==2022.1.0
s3transfer==0.5.0
semver==2.13.0
six==1.16.0
smart-open==5.2.1
snowflake-connector-python==2.4.6
snowflake-rdp-broker==2.5.5
toml==0.10.2
urllib3==1.26.8
wrapt==1.13.3
yarl==1.7.2

Any help is much appreciated! Not sure if this is a bug

[martindurant]

This apparently happening deep within aiobotocore, I would first check that your versions of aiobotocore and botocore are compatible (they are pinned one-to-one), and have a look if there is something similar in the aiobotocore issues.

[mavalur]

has there been any further findings on this one. Apparently, this started showing up in the last few days.

[jowparks]

Thanks @martindurant your comment led me to a temp fix at least.

I am guessing for anyone else having this problem these would fix it for you if you pin your dependencies:

s3fs==2022.1.0
aiobotocore==2.1.0
awscli==1.22.24
boto3==1.20.24

I fixed it for now by pinning basically everything to correct versions, note that it wasn't enough to pin version of aiobotocore, it also seemed like the issue was somehow related to awscli being a compatible version.

My pinned dependencies that worked were:

snowflake-rdp-broker
s3fs==2022.1.0
smart-open==5.2.1
aiobotocore==2.1.0
awscli==1.22.24
boto3==1.20.24
pytest==6.2.4
pytest-mock==3.6.1

Note that the snowflake-rdp-broker (an internal package) had its own dependencies for awscli/boto3 and that was likely causing the issue. Also, I have to use the --use-deprecated=legacy-resolver flag during pip install so that the dependencies don't get downgraded or resolved. sigh python dependency fun.

The working version I have tested has these packages for pip freeze:

aiobotocore==2.1.0
aiohttp==3.8.1
aioitertools==0.8.0
aiosignal==1.2.0
asn1crypto==1.4.0
async-timeout==4.0.2
attrs==21.4.0
awscli==1.22.24
boto3==1.20.24
botocore==1.23.24
certifi==2020.12.5
cffi==1.15.0
charset-normalizer==2.0.11
colorama==0.4.3
cryptography==36.0.1
docutils==0.15.2
frozenlist==1.3.0
fsspec==2022.1.0
idna==3.3
iniconfig==1.1.1
jmespath==0.10.0
multidict==6.0.2
numpy==1.22.2
oscrypto==1.2.1
packaging==21.3
pandas==1.2.5
pluggy==0.13.1
py==1.11.0
pyarrow==6.0.0
pyasn1==0.4.8
pycparser==2.21
pycryptodomex==3.14.0
PyJWT==2.3.0
pyOpenSSL==21.0.0
pyparsing==3.0.7
pytest==6.2.4
pytest-mock==3.6.1
python-dateutil==2.8.2
pytz==2021.3
PyYAML==5.4.1
requests==2.27.1
rsa==4.7.2
s3fs==2022.1.0
s3transfer==0.5.1
semver==2.13.0
six==1.16.0
smart-open==5.2.1
snowflake-connector-python==2.7.4
snowflake-rdp-broker==2.5.7
toml==0.10.2
urllib3==1.26.8
wrapt==1.13.3
yarl==1.7.2

[mavalur]

Pinning to the following version worked. Thank you @jowparks .

s3fs==2022.1.0 aiobotocore==2.1.0 boto3==1.20.24 botocore==1.23.24 pyarrow>6.0.0,<=7.0.0

[jrasky]

Just pinning botocore<=1.23.46 worked for me

[apiszcz]

+1

[Zyiqin-Miranda]

+1