cachetools KeyError from auth/access_token_client.py

[chazmcgarvey]

Describe the bug

We occasionally catch KeyError exceptions raised from sp_api/auth/access_token_client.py:50 when calling Reports.create_report.

0: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 97"
1: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 70"
2: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 418"
3: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 113"
4: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 366"
5: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 494"
6: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 79"
7: "/venv/lib/python3.12/site-packages/cachetools/__init__.py: 423"
8: "/venv/lib/python3.12/site-packages/sp_api/auth/access_token_client.py: 50"
9: "/venv/lib/python3.12/site-packages/sp_api/base/client.py: 70"
10: "/venv/lib/python3.12/site-packages/sp_api/base/client.py: 63"
11: "/venv/lib/python3.12/site-packages/sp_api/base/client.py: 97"
12: "/venv/lib/python3.12/site-packages/sp_api/api/reports/reports.py: 113"
13: "/venv/lib/python3.12/site-packages/sp_api/base/helpers.py: 21"

To Reproduce

I'm not sure, yet, what conditions cause the KeyError.

Expected behavior

Handle the KeyError in the library. If there actually is an exceptional condition, raise a more specific exception.

Desktop (please complete the following information):

• OS: Linux (cloud)

Additional context

• Using python-amazon-sp-api 1.4.0.
• Using cachetools 5.3.2.
• Using python 3.12.2.
• We're calling Reports.create_report from another thread (via asyncio.to_thread).

[saleweaver]

Is this a long running environment that's sending the requests to the thread?

[chazmcgarvey]

Is this a long running environment that's sending the requests to the thread?

The process might run for several hours. Never more than a day. The environment itself is ephemeral, lasting as long as the process.

[saleweaver]

That's the problem. The access token is valid for an hour, and if it gets deleted just when you're requesting, the key doesn't exist. I'll have a look if there's an easy fix, I haven't used long running tasks for this in years.

[chazmcgarvey]

Thanks. If the condition can be handled transparently, great, but even if there was just a more specific exception raised we could catch and know to just retry, that'd work.

[saleweaver]

Yeah, going to identify and clarify it at least 😄

[nonfatalexec]

I think I'm getting the same error. I upgraded my OS which upgraded Python then this started happening. It's 100% reproducible. It was working everyday for the previous year before I upgraded. I tried removing pycache in case that was related, but it made no difference.

$ pip list
Package              Version
-------------------- -----------
boto3                1.34.79
botocore             1.34.79
cachetools           5.3.3
certifi              2024.2.2
charset-normalizer   3.3.2
confuse              2.0.1
idna                 3.6
jmespath             1.0.1
pip                  23.2.1
python-amazon-sp-api 1.4.3
python-dateutil      2.9.0.post0
PyYAML               6.0.1
requests             2.31.0
s3transfer           0.10.1
setuptools           67.7.2
six                  1.16.0
urllib3              1.26.18
wheel                0.40.0

  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/cachetools/__init__.py", line 68, in __getitem__
    return self.__data[key]
KeyError: 'access_token_57823aada8f070e62b53ef9039dbabf0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/auth/access_token_client.py", line 46, in get_auth
    access_token = cache[cache_key]
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/cachetools/__init__.py", line 418, in __getitem__
    return cache_getitem(self, key)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/cachetools/__init__.py", line 70, in __getitem__
    return self.__missing__(key)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/cachetools/__init__.py", line 97, in __missing__
    raise KeyError(key)
KeyError: 'access_token_57823aada8f070e62b53ef9039dbabf0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./get_inventory.py", line 25, in <module>
    report_request = reports_api.create_report(reportType="GET_FLAT_FILE_OPEN_LISTINGS_DATA", dataStartTime=str(date.today())+"T04:00:00", dataEndTime=str(date.today())+"T04:00:00")
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/base/helpers.py", line 21, in wrapper
    return function(*args, **kwargs)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/api/reports/reports.py", line 113, in create_report
    return self._request(kwargs.pop('path'), data=kwargs)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/base/client.py", line 97, in _request
    headers=headers or self.headers,
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/base/client.py", line 63, in headers
    'x-amz-access-token': self.restricted_data_token or self.auth.access_token,
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/base/client.py", line 70, in auth
    return self._auth.get_auth()
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/auth/access_token_client.py", line 49, in get_auth
    access_token = self._request(request_url, self.data, self.headers)
  File "/home/gerald/amazon-sp-api/lib/python3.8/site-packages/sp_api/auth/access_token_client.py", line 35, in _request
    raise AuthorizationError(error_code, error_message, response.status_code)
sp_api.auth.exceptions.AuthorizationError: ('invalid_client', 'Client authentication failed', 401)