aiobotocore resolve RefreshableCredentials.token issue

we currently disable the token property because it needs to be an async call. See: https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/credentials.py#L259

This is called in places like SigV4Auth._modify_request_before_signing which is called by SigV4Auth.add_auth. I haven't check upstream who calls this yet

Mar 02 '22 01:03 thehesiod

@terrycain just ran into this, any thoughts? Normally this just means overriding a ton more methods :(

Mar 02 '22 01:03 thehesiod

Not come across it yet, yeah it'll probably end up in overriding lots more stuff :(

Mar 02 '22 10:03 terricain

@terrycain how did you create your avatar icon btw? very nice!

Mar 02 '22 17:03 thehesiod

ty lol. Not a clue, got given it by a previous company :D

Mar 03 '22 23:03 terricain

https://github.com/boto/botocore/blob/e720eefba94963f373b3ff7c888a89bea06cd4a1/botocore/auth.py#L886

From the looks of it, SigV4Auth is subclassed by S3SigV4Auth which calls _modify_request_before_signing which then calls the superclass's _modify_request_before_signing which calls self.credentials.token.

I saw a github issue somewhere saying v4 is required for eu-central-1 so I managed to trigger this using the latest boto3 doing a put_object on a bucket there.

Current call stack for a put_object looks roughly like:

"s3_client.put_object"
ClientCreator._create_api_method._api_call
BaseClient._make_api_call
BaseClient._make_request
Endpoint._make_request
Endpoint._send_request
Endpoint.create_request
...some self._event_emitter.emit bs
RequestSigner.handler
RequestSigner.sign
SigV4Auth.add_auth
S3SigV4Auth._modify_request_before_signing
SigV4Auth._modify_request_before_signing

iirc we've asyncified most if not all of the main emit logic and general request signing, so it might not be too bad

Mar 03 '22 23:03 terricain

Looking a bit more into this before I sleep.

https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/signers.py#L70 - We'll probably need to hijack AUTH_TYPE_MAPS and substitute our own async variants then asyncify https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/signers.py#L63

Then in the signers, asyncify add_auth and _modify_request_before_signing and in there call await get_frozen_credentials

All in all my sleep deprived brain doesn't seem to think there'll be much to patch. If I remember I'll have a crack at it Friday evening. Been too long since I've been active in this area :D

Mar 04 '22 00:03 terricain

wow didn't expect so much time dedicated to this, hehe, awesome. I ran into this because we're migrating to opensearch which apparently requires SigV4Auth, just created some aiohttp helpers for that, maybe will make some time to make a module for that. I'd post it on a gist but then someone will snarf it before I create a module ;)

Mar 04 '22 00:03 thehesiod

Ok from what I can see. The only place SigV4Auth is instantiated is in the AUTH_TYPE_MAPS dict in auth.py, and the only place it's values are accessed would be in signers.py, which we've supplied the credentials as get_frozen_credentials() so calling self.credentials.token should be safe as it should be an instance of ReadOnlyCredentials.

In theory if calling get_frozen_credentials on one of the credential providers doesn't return a ReadOnlyCredentials object then it would cause issues. Don't suppose you can send me some info on how to recreate it.

Mar 05 '22 20:03 terricain