boto
Refactoring HMAC signers in botocore/auth.py to support Yubikey-based signing
I have an initial implementation of a package for storing AWS access keys on Yubikey devices (https://github.com/pyauth/exile) and having the Yubikey sign AWS requests. This allows me to store the secret key for any given key ID on a Yubikey instead of in the home directory, protecting it in the YubiKey (which acts as an HSM, so theft of home directory contents does not compromise the key), and optionally further requiring user interaction (pressing a button on the key) to sign the request.
To integrate this into botocore, I currently have to monkey-patch signers in botocore/auth.py. The first step to not needing this monkey-patching is to factor out HMAC signers so instead of taking the literal secret HMAC key and string to sign, they can take a (botocore.credentials.Credentials, string_to_sign) tuple.
After that I can make a botocore.credentials.CredentialProvider subclass that would indicate that the credential is actually in the Yubikey, and think of how to structure the rest of the plumbing.
Does this sound reasonable? If so, I can get started on the first PR for factoring out the HMAC signers.
Hey that sounds really cool! If you want to include a custom credential provider for botocore there are two ways to go about it. The first way is to just put it in the provider chain:
from botocore.session import Session
session = Session()
provider_chain = session.get_component('credential_provider')
provider_chain.insert_after('config-file', my_provider)
Another way, which would also allow for supporting the CLI and several other SDKs, would be to make use of the process provider. There's some instructions on how to do that in the awsprocesscreds repo, which is itself an implementation of that provider.
So you would set something like this in ~/.aws/config:
[profile yubikey]
region = us-west-2
credential_process = exile
When the CLI or SDK needs credentials it would invoke that process. There you handle prompting and communicating with the yubikey. You would then return credentials in json format over stdout.
Thanks for the pointers, that's great, I will look into that when writing a credential provider.
What I was trying to say is that there's a fundamental issue in botocore/auth.py, that I'll have to address before I build a custom credential provider. The signers in auth.py assume that the credential secret key is available by value. That's not the case with the Yubikey - the secret key is sequestered in the device, and we can only ask the device to perform HMAC using the key ID. So the places where botocore/auth.py signers call hmac() would need to be factored out into an HMAC helper, which would take the whole Credential object instead of the secret key value, and look at the type of Credential. If it has a literal secret key, then it would call hmac() directly. If it's a Yubikey credential, then it would call out to the Yubikey with the key ID, and return the HMAC computed by the Yubikey.
I'll get started on a PR for that.
Alternatively, the protocol could be to have a .sign() method on the Credential object. The default .sign() method would just return hmac(self.secret, string_to_sign), and the custom Credentials could do something else.
Ah, I see. Interesting. This would be somewhat complicated by the fact that for sigv4 you prepend "AWS4" to the key, which you don't do for sigv2. In practice this only matters if you're using sigv2 on S3 or if you're using simpledb, but that ends up being a lot of people. A new signing version may do something similar, like "AWS5". Based on my cursory skim of their docs it looks like that means you would need to store two different 'hmac keys' and switch between them based on signature version.
Yeah, thanks for pointing that out. That's exactly what I did - I store multiple versions of each key, using an identifier that includes both the key ID and the signature/signer version (so far just two, SigV4 and HmacV1. Then I key into it based on which signer is calling.
@JordonPhillips, I've made good progress on this plan. If we merge #1723, I can use pluggable yubikey based authentication with this code, which is still a bit verbose, but does what I want. Next, I'm going to look at your advice to use credential_process to get profile-based configuration working with less boilerplate and available in the CLI.
from botocore.session import Session
from botocore.credentials import SigningCredentials, EnvProvider, SharedCredentialProvider
from botocore.exceptions import ConfigNotFound
from exile import YKOATH
from exile.exceptions import YKOATHError
class YKCredentials(SigningCredentials):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._ykoath = YKOATH()
def get_frozen_credentials(self):
return self
def hmac_sign(self, msg, hex=False, version=None):
key_name = "exile-{}-SigV4".format(self.access_key)
return self._ykoath.calculate(key_name, msg.encode(), want_truncated_response=False)
class YKCredentialProvider:
def _access_key_found_on_device(self, access_key):
try:
yubikey = YKOATH()
credentials = [c.name for c in list(yubikey)]
if "exile-{}-SigV4".format(access_key) in credentials:
return True
except YKOATHError:
return None
class YKEnvCredentialProvider(EnvProvider, YKCredentialProvider):
METHOD = 'yubikey-env'
def load(self):
if self._mapping['access_key'] in self.environ:
access_key = self.environ.get(self._mapping['access_key'])
if self._access_key_found_on_device(access_key):
return YKCredentials(access_key=access_key, secret_key=None)
class YKSharedCredentialProvider(SharedCredentialProvider, YKCredentialProvider):
METHOD = 'yubikey-shared-credentials-file'
def load(self):
try:
available_creds = self._ini_parser(self._creds_filename)
except ConfigNotFound:
return None
if self._profile_name in available_creds:
config = available_creds[self._profile_name]
if self.ACCESS_KEY in config:
access_key, secret_key = self._extract_creds_from_mapping(config, self.ACCESS_KEY, self.SECRET_KEY)
if self._access_key_found_on_device(access_key):
return YKCredentials(access_key=access_key, secret_key=None)
session = Session()
provider_chain = session.get_component('credential_provider')
yk_env_provider = YKEnvCredentialProvider()
provider_chain.insert_before('env', yk_env_provider)
credential_file = session.get_config_variable('credentials_file')
profile_name = session.get_config_variable('profile') or 'default'
yk_profile_provider = YKSharedCredentialProvider(creds_filename=credential_file, profile_name=profile_name)
provider_chain.insert_before('shared-credentials-file', yk_profile_provider)
print(session.create_client('sts').get_caller_identity())
I would like to see this feature added. Seems like the perfect way to prevent theft of Access Keys.
I closed the PR that was attached to this issue because signing has changed a bit in the time since this issue was opened. For example, old signature versions are no longer supported and we'd want to rethink the implementation before moving forward. I will leave this issue open for now in case there is any additional discussion on this topic.
