aioaws

aioaws copied to clipboard

Description

Resolves samuelcolvin/aioaws#16

Thanks for this project. It’s refreshing to see an alternative to the other "bloated, opaque" AWS tools out there.

This PR will add basic AWS session token support to aioaws. AWS session tokens are used when an AWS resource obtains temporary security credentials.

The authorization flow for temporary security credentials commonly works like this:

• IAM roles, such as service-linked roles or Lambda execution roles, are set up and linked to infrastructure resources. These roles can have two kinds of IAM policies attached: resource-based and identity-based policies.
  • Identity-based policies define interactions with other resources on AWS.
  • A resource-based policy called a "role trust policy" defines how the role can be assumed.
• The AWS runtime (Fargate, Lambda, etc) requests authorization to use the role by calling the STS AssumeRole API.
• If the requesting entity has permissions to assume the role, STS responds with temporary security credentials that have permissions based on the identity-based policies associated with the IAM role.
• The AWS runtime stores the temporary security credentials, typically by setting environment variables:
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_SESSION_TOKEN
• AWS API calls with temporary credentials must include the session token, as mentioned in samuelcolvin/aioaws#16.
• The AWS runtime will typically rotate the temporary security credentials before they expire.

Changes

Source code

• Add aws_session_token: str model fields
  • aioaws._types.S3ConfigProtocol
  • aioaws.s3.S3Config
• Add aioaws.core.AwsClient.aws_session_token attribute
• Update aioaws.core.AwsClient._auth_headers to add session token into signature
• For S3 downloads, add X-Amz-Security-Token param to aioaws.core.AwsClient.add_signed_download_params
• For S3 uploads, allow X-Amz-Security-Token as an extra condition in aioaws.core.AwsClient.upload_extra_conditions, then add to upload signature using aioaws.core.AwsClient.signed_upload_fields

Tests

• Make S3 bucket name configurable in tests: S3 bucket names are globally unique, so other contributors will need to create buckets with different names. They can then set ${{ secrets.TEST_AWS_S3_BUCKET_NAME }} in GitHub Secrets to use the bucket name in GitHub Actions. Defaults to the previous hard-coded value (aioaws-testing).
• Make SES email address configurable in tests: As with S3 bucket names, enabling configuration of the email address allows contributors to run tests on their forks and in their AWS accounts, by setting ${{ secrets.TEST_AWS_SES_ADDRESS }} in GitHub Secrets. Defaults to the previous hard-coded value ([email protected]).
• Add workflow_dispatch trigger to GitHub Actions workflow: Enables workflow to be run manually, from any branch. Useful for testing code changes on feature branches.

IAM policies

• Add role trust policy document aws_test_role_permissions.json
  • sts:AssumeRole allows GitHub Actions to obtain temporary security credentials
  • sts:TagSession allows GitHub Actions to tag the temporary security credentials with metadata as described here and here

Usage

AWS_SESSION_TOKEN can be added with the S3Config.aws_session_token attribute.

S3Config('<access key>', '<secret key>', '<region>', 'my_bucket_name.com', os.getenv('AWS_SESSION_TOKEN', ''))

Maintainer TODOs

There are a few steps needed to enable AssumeRole for testing. This may be confusing, so I'm happy to help with setup as needed. We can also do this with OpenID Connect if you prefer.

• Update the new role trust policy file aws_test_role_permissions.json with:
  • <AWS_ACCOUNT_NUMBER>: Your account number
  • <IAM_USERNAME>: The name of the IAM user that owns the access key used for testing
• Create an IAM role for the user to assume (same user that owns the access key used for testing), using aws_test_role_permissions.json as the role trust policy
• Create a customer-managed identity-based policy using aws_test_key_permissions.json, if you haven't already
• Attach the managed policy from aws_test_key_permissions.json to the role

AWS CLI steps look like this:

cd aioaws

# copy your account number (replace fx with jq or other JSON parser as needed)
AWS_ACCOUNT_NUMBER=$(aws sts get-caller-identity | fx .Account)

# add account number and username to role trust policy (this is GNU sed syntax)
gsed -Ei "s|<AWS_ACCOUNT_NUMBER>|$AWS_ACCOUNT_NUMBER|g" aws_test_role_permissions.json
gsed -Ei "s|<IAM_USERNAME>|aioaws|g" aws_test_role_permissions.json

# create IAM role using the role trust policy document
aws iam create-role --role-name aioaws-testing-role --assume-role-policy-document file://aws_test_role_permissions.json

# create a customer-managed identity-based policy if you haven't already
POLICY_ARN=$(aws iam create-policy --policy-name aioaws-testing --policy-document file://aws_test_key_permissions.json | fx .Policy.Arn)

# attach the policy to the role
aws iam attach-role-policy --policy-arn $POLICY_ARN --role-name aioaws-testing-role

The end result is that the AWS access key can be used to assume the role and obtain temporary credentials. The temporary credentials have the same IAM policy as the regular access key, so tests can be parametrized accordingly.

Notes

Session token expiration

aioaws clients do not automatically rotate temporary credentials. Developers are responsible for updating client attributes or instantiating new clients when temporary credentials expire.

Token expiration should be taken into account when generating S3 presigned URLs. As explained in the docs, "If you created a presigned URL using a temporary token, then the URL expires when the token expires, even if the URL was created with a later expiration time."

Other credential sources

There are several other ways to source credentials (see the AWS IAM docs, AWS CLI docs, and Boto3 docs). This project only handles AWS access keys and session tokens.

For CI/CD, GitHub Actions supports OpenID Connect ("OIDC"), which allows workflows to authenticate with AWS without having to store access keys in GitHub Secrets.

looking good, but we need to get tests to work before I can review properly.

Thanks for your review! I would welcome any further comments or suggestions.

really not sure why tests didn't run for your two pull requests, I just got a friend to try and PRs from external contributors are working #26

really not sure why tests didn't run for your two pull requests, I just got a friend to try and PRs from external contributors are working #26

Tests appear to be running after merging the master branch.

You'll need to set up a role in order to test session tokens. See the "Maintainer TODOs" section above. Once you get it set up, runs should succeed like this.

AWS CLI steps would look like this:

cd aioaws

# copy your account number (replace fx with jq or other JSON parser as needed)
AWS_ACCOUNT_NUMBER=$(aws sts get-caller-identity | fx .Account)

# add account number and username to role trust policy (this is GNU sed syntax)
gsed -Ei "s|<AWS_ACCOUNT_NUMBER>|$AWS_ACCOUNT_NUMBER|g" aws_test_role_permissions.json
gsed -Ei "s|<IAM_USERNAME>|aioaws|g" aws_test_role_permissions.json

# create IAM role using the role trust policy document
aws iam create-role --role-name aioaws-testing-role --assume-role-policy-document file://aws_test_role_permissions.json

# create a customer-managed identity-based policy if you haven't already
POLICY_ARN=$(aws iam create-policy --policy-name aioaws-testing --policy-document file://aws_test_key_permissions.json | fx .Policy.Arn)

# attach the policy to the role
aws iam attach-role-policy --policy-arn $POLICY_ARN --role-name aioaws-testing-role

It's not pull_request: {} since #26 ran tests.

I'll look into the role changes next week.

can you remove the changes to .github/workflows/ci.yml they seem to be causing the PR to fail.

can you remove the changes to .github/workflows/ci.yml they seem to be causing the PR to fail.

Again, you need to create a role in your AWS account for the updated tests to work.

Sorry, I'll do that.

Closing off this PR because it hasn't moved forward in about a year, and I no longer have a use case for this package.

Sorry I didn't get around to completing this, and thanks for the contribution. 🙏