boto3
boto3 copied to clipboard
boto
refreshing sts role assumption credentials for long running operations
boto3 looks like it wires in credential refreshing when its using instance roles by default. The issue is how how to setup the same behavior when doing sts role assumption, to refresh credentials (i'm initially starting with an instance role). Atm the only way to pass the sts assumed role credentials is in via explicitly creating a session/client with key/secret/token, but that results in the creation of a default Credentials class without the refresh behavior. There doesn't seem to be a clean way to provide a RefreshableCredential acquired with sts token assumption to a client.
by clean way, i mean not poking at botocore session internals for example, this is what i'm currently doing to make this work https://gist.github.com/kapilt/ac8e222081f63ba64e93
I'm not a core dev but this is how I'm dealing with AWS access/secret keys, regions, etc: https://github.com/russellballestrini/botoform/blob/master/botoform/util.py#L18-L87
Edit: This class makes it possible to switch between the AWS config profile/region programatically.
There is an AssumeRoleProvider for credentials in Botocore that might help here: https://github.com/boto/botocore/blob/develop/botocore/credentials.py#L589. I'll try to get more information on this and see if I can provide an example.
Please also feel free to chime in to the related discussion happening on Botocore about how we can improve assume role support: https://github.com/boto/botocore/issues/761
@russellballestrini do you have a gist of consuming that code? I want to use it for s3 client for downloading and uploading files, a process that goes on for hours if not days.
EDIT: I don't see how the refresh is being auto refreshed.
@BardiaAfshin the code that I linked was never tested with "assuming a role". That class just makes it possible to switch between the AWS config profile/region programatically.
Sorry about that.
Anybody have any clue how to get cert refresh with assume_role_with_saml?
Figured it out.
https://gist.github.com/JoeyG1973/69ae503f67ff7f07b498b2e53226e206
@kapilt Thank you! Your method works for me as well. In my case, the client application gets cognito credentials (expiring in 1 hr) and does an s3 upload that may run for hours. I used assume_role_with_web_identity() in the refresh, with WebIdentityToken = the cognito token
It's been more than a year since the last activity on this thread. Have people found a better workaround than @kapilt 's? Thanks.
@BardiaAfshin, I have the same issue with S3. Any chance you found a solution?
Found a solution which only uses the public api of Boto3 and Botocore. Gist can be found here. Feedback = welcome.
https://github.com/monkeysecurity/botor is really close to doing what you need. There is at least one additional incarnation of this https://github.com/Netflix-Skunkworks/cloudaux, but I found that some of them just refresh the STS token and don't cache the connections. It's not hard to test and fix. It is a python decorator that you wrap all your AWS SDK calls with. It creates tokens and refreshes them when they are almost expired. It also caches connections. I made minor modifications so that it works with or without assumed role. It's not seamless, but it it requires no changes to core AWS SDK and it is working well for our needs.
I have the same problem. In my case I would like to transfer files to S3 using AssumeRole. Since transfer stops when it expires, you need to update your credentials. Is there a better way to update my credentials without stopping the transfer?
I strongly suggest this issue be CLOSED - it is a non issue. i.e. chose the tool for the job. AWS solve this use case, all you need to do is use an instance profile and assign the role to it, Amazon will keep the credentials alive for you.
If you are not using a host managed by Amazon, and the 12 hour limit on STS Assume Role is too short for your use case, then STS Assume Role is not the tool for the job, you will have to do these problematic hacks to make the square peg fit in your round hole, not smart.
The appropriate solution might be using IAM keyId and secret, or federated through Cognito, or something else. Bottom line, if you need more than a 12 hour session for a long running use case STS Assume Role is an anti-pattern and should be avoided.
Note to maintainers; do not change botocore or boto3 to appease such feature requests that any AWS solution archetect, and all AWS documentation and materials discourage. Point these issues to the appropriate AWS documentation to help the educational issues, and suggest the Labs and AWS certified developer associate / professional exam materials.
There are many legitimate use cases for this, please don't make assumptions and try to impose them on others without understanding their use cases. sts role assumption is needed for crossing account boundaries even from an instance with profile, and long running operations will need refresh. static iam creds are much more an anti pattern from a security perspective. I think there is a bit more support in botocore for refresh, but some assembly still required, the cleanest I've seen is this one https://github.com/boto/botocore/issues/761#issuecomment-426037853
incorrect, iam key and secret is not poor security practice at all. As an Information Security professional the security risks are poor key management. sts assumerole also requires iam key and secret that has permissions for sts and iam actions when assuming outside an aws managed compute environment therefore the same inherent risks apply in both cases. there is actually more risk when developers poorly implement hacks to force sts to work outside its 12 hour limit.
Also all code samples provided, and the feature request itself, do not address the cross account boundary use case, it is clearly a request to keep alive an assumed role for longer running processes than 12hr limitation. A good developer may find a solution and contribute upstream but a great developer solves the root problem for their use case.
Security risks aside, as this is a pure right-tool-for-the-job statement, and preserving the library from being compromised. As developers must seek clarity from those with knowledge before writing poor code that compromises the project. I impose nothing more than as boto maintainers to consultant AWS about this feature request, and I simply explain the facts to consider.
I'm looking for a solution to this as well. I have a Lambda which assumes roles in other accounts.
boto3.Session() can take a profile_name but this requires ~/.aws/config/~/.aws/credentials files to work they these files don't exist in Lambda.
I think it would be enough if boto3 provided a way to configure profiles programmatically. Maybe that's already there? I haven't found it.
Just found this.
Check out get_role_session from boto3_extensions ... (permalink at time of commenting)
This is exactly what I was mentioning. They're monkey-patching the configuration in memory instead of using a file. This is also a hack. There really should be a way to programmatically specify a profile.
We use role chaining which AWS sets an unchangeable hard session limit of 1 hour on, which makes credential refreshing necessary.
the only real question is; should a framework be dogmatic? should a framework transparently manage auth keep-alive for the user? should a framework circumvent the vendors own restrictions that are based on security best-practices?
The answer to all of those I hope keeps these rubbish insecure ideas out of boto3 and let the users who want to be insecure do their own hacks that should not be in a framework like boto3.
@chrisdlangton when using an IAM role assigned to an EC2 instance this framework does exactly what you mention. It manages keep-alive for the user. It is completely transparent. Refreshing tokens using that 169.254.169.254 metadata service.
Here we're asking that the functionality be extended to support using an assumed role as well, not just the one assigned to the instance.
I would argue that
- Long running jobs on an EC2 are a valid use case and not against security best-practices.
- Long running jobs on an EC2 which assume a role are a valid use case and not against security best-practices.
If you have any links / documentation saying otherwise, I'd be happy to read them.
@ericfrederich why be so petty as to make it sound like it was my opinion in context to what you describe? I responded to @PSiAU commenting on the 1hr session limit, don't go generalizing my response to the 1hr limitation to make your own arguments sound better. Come up with your own value proposition to make the case to add bespoke hacks to boto3 that fit into your model of how frameworks should work and don't go generalizing other peoples responses to specific comments you didn't even make.
p.s. if you're looking for a link about the 1hr session limit it's clear you're not educated or capable of finding the knowledge (ignoring you asked me and I didn't even make the point)
Also +1, my team is facing a similar issue and ended up using the gist @kapilt provided. Assumed roles are generally the solution AWS Support recommends when performing cross account operations, and there are scenarios you need these sessions refreshed. The fact that RefreshableCredentials exist at all speaks to the admission of the validity of that use case. Also I'd argue periodic session refreshing is more secure than hard coded AKey/SKey values. If the later is leaked once you have a major security issue until the keys are changed manually, depending on your setup that may require redeployment of software. If session details created from assumed role are leaked, you have a major security issue until they expire or you revoke them. The argument that refreshing them is some how insecure, especially considering that this is very similar to how the credentials for instance profiles are rotated, doesn't make a lot of sense to me.
Well reasoned and said but in general, Don’t feed the trolls
On Wed, May 22, 2019 at 3:51 PM jimcarreer [email protected] wrote:
Also +1, my team is facing a similar issue and ended up using the gist @kapilt https://github.com/kapilt provided. Assumed roles are generally the solution AWS Support recommends when performing cross account operations, and there are scenarios you need these sessions refreshed. The fact that RefreshableCredentials exist as at all speaks to the admission of the validity of that use case. Also I'd argue periodic session refreshing is more secure than hard coded AKey/SKey values. If the later is leaked once you have a major security issue for until the keys are changed manually, depending on your setup that may require redeployment of software. If a session details created from assumed role are leaked, you have a major security issue until they expire or you revoke them. The argument that refreshing them is some how insecure, especially considering that this is very similar to how the credentials for instance profiles are rotated, doesn't make a lot of sense to me.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/boto/boto3/issues/443?email_source=notifications&email_token=AAAFJET7DMH74ZJYNOQ472DPWWW3FA5CNFSM4BYZZWHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWAJXAQ#issuecomment-494967682, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAFJES3LZOQU7YVFKSPCD3PWWW3FANCNFSM4BYZZWHA .
@kapilt so a comment that only name calls and confirmation bias callouts is called what?
On-topic, the most recent example provide here by @PSiAU has an AWS restriction of 1hr. If you do not understand the security for that choice, please do not assume that you are correct to circumvent the security controls by promoting boto3 to add your preferred solution to circumvent a security control.
I'm not arguing for imposing any restrictions on anyone, I am arguing to avoid boto3 enabling users who want to circumvent security controls.
case-in-point I have a separate feature request and possible PR to add Perfect Forward Secrecy to boto3.
Respect AWS security controls should not be circumvented. Period.
@chrisdlangton, recapping
- sts session time out is configurable from 1-12 hours, it is not 1 hour, that is merely a default. https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html
- static credentials for iam users are a security anti-pattern, many organizations disable iam users api keys entirely.
- control of whether or not session renewal is possible is controlled via iam permissions, ie the ability of the iam role to invoke AssumeRole on itself, not by the sdks
- attempting to prevent sdks from offering ergonomic interfaces in the name of security, when the underlying capability is available and in common use, is arguing for security through obscurity while misunderstanding the real security controls here are iam based not sdk.
there are many legitimate application use cases for this, when performing any long running operation that needs to cross role or account boundaries. the job of sdks is to enable api usage and application development, if there are common tasks for applications around api usage, rather than having everyone reimplement the same, the better outcome is to have the sdk support that functionality directly.
