aws-load-balancer-controller
aws-load-balancer-controller copied to clipboard
kubernetes-sigs
TargetGroupBindings can now manipulate target groups from different aws accounts
Issue https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/3634
Description
Now, every time a TargetGroupBinding has the alb.ingress.kubernetes.io/IamRoleArnToAssume (and optionally alb.ingress.kubernetes.io/AssumeRoleExternalId) annotations, we assume it before interacting to it (actually, the assume role operation is cached).
In real life, I changed:
- targetgroupbinding_validator
- targetgroupbinding_mutator
- target manager (which has
RegisterTargets/DeregisterTargets/ListTargets
This way we can easily manipulate target groups from target group bindings for every single AWS account in the world, as long as there is a role we can assume. In order to prevent the confused deputy problem, external id is supported.
The way this was implemented was actually way simpler than initially imagined:
the service.elbv2Client object now supports a new method called AssumeRole which returns service.elbv2Client on the specified role. The catch is that if AssumeRole is called with a blank role, the object returns itself and life goes on.
In other words, every time we call some method from elbv2Client from something related to target groups, we also call AssumeRole and that's basically it.
Caching the assumed role was also trivial, a simple map that is stored on cloud.go
I decided to have cloud.go manage the caching becuase in the future it could assume role for other objects if needed and all the assume role logic would be in the same place.
The only unfortunate change that needed to be made is that aws.Cloud became services.Cloud else golang would complain about circular dependency :(
This is what the logs look like:
{"level":"info","ts":"2024-05-13T03:24:31Z","msg":"awsCloud","method":"GetAssumedRoleELBV2","AssumeRoleArn":"arn:aws:iam::7550XXXXXXXX:role/alb-controller-policy-to-impersonate","externalId":"XXXXXX"}
{"level":"info","ts":"2024-05-13T03:24:33Z","msg":"registering endpoints using the targetGroup's vpcID vpc-00XXXXXXXXXXXXXXX which is different from the cluster's vpcID vpc-04XXXXXXXXXXXXXXX"}
{"level":"info","ts":"2024-05-13T03:24:33Z","msg":"registering targets","arn":"arn:aws:elasticloadbalancing:us-east-1:7550XXXXXXXX:targetgroup/helloworld-8080-a8054/5741da332459b560","targets":[{"AvailabilityZone":"all","Id":"10.244.0.22","Port":8080}]}
{"level":"info","ts":"2024-05-13T03:24:34Z","msg":"registered targets","arn":"arn:aws:elasticloadbalancing:us-east-1:7550XXXXXXXX:targetgroup/helloworld-8080-a8054/5741da332459b560"}
Documentation and tests were properly updated in this PR
Checklist
- [ ] Added tests that cover your change (if possible)
- [X] Added/modified documentation as required (such as the
README.md, or thedocsdirectory) - [X] Manually tested
- [X] Made sure the title of the PR is a good description that can go into the release notes
BONUS POINTS checklist: complete for good vibes and maybe prizes?! :exploding_head:
- [ ] Backfilled missing tests for code in same general area :tada:
- [X] Refactored something and made the world a better place :star2:
A working amd64 container with this code is available here: marcosdiez/aws-load-balancer-controller:v2.7.2-m2
update: on 2024-09-26 I rebased the code. The new container image is marcosdiez/aws-load-balancer-controller:v2.8.3-m1
Hi @marcosdiez. Thanks for your PR.
I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@shraddhabang I believe my code is ready. Feel free to review at will! Thank you.
Also, if you don't want the CRDs to be changed, here: https://github.com/kubernetes-sigs/aws-load-balancer-controller/pull/3691/commits/126c9c82e53806d8b3755b481539e88bac629222
they are untouched.
Thank you @marcosdiez for this PR. This looks excellent. I will take a look and test it out. And would also ask our team to review it. Please note that we will have to go through our formal security process to review this. We will add this to our next minor release v2.9.0. Please feel free to reach out to us.
Hi @shraddhabang ,
We will add this to our next minor release v2.9.0
Could you please share us when the v2.9.0 would release? We are looking forward to this feature.
Hi, @johngmyers @M00nF1sh @shraddhabang
We will add this to our next minor release v2.9.0.
Does anyone know the release schedule for v2.9.0 ?
I just rebased and merged my code. There is also a container available here: marcosdiez/aws-load-balancer-controller:v2.8.3-m1
Small update: the old docker container was broken. Please use marcosdiez/aws-load-balancer-controller:v2.8.3-m2 instead (it was not taking ExternalID into consideration). The code has been rebased and updated again.
Hey @marcosdiez, 10x for the great work
i tried your image (marcosdiez/aws-load-balancer-controller:v2.8.3-m2) and it works great :) the targets were synced to the remote account ALB targetgroup :)
FYI, at first i didn't provided the necessary permissions and the pod crashed, with a good error message :) i think that the error message is enough, it will be great if the pod wont crash :)
waiting for a Merge to main :)
@shraddhabang any plans on merging this ? It took me 8 hours for this last rebase....
@shraddhabang appreciate if you can speed up this merge/release. There's significant work done on it now.
I am so desperate for this feature. I get excited when there is a new release but then see that it is not included :cry:
Amazing. We have been waiting for this feature for a long time. Could you approve and merge it?
I've been eagerly awaiting this feature for a long time. Please approve and merge it as soon as possible. This feature is crucial for the efficient delivery of my infrastructure, as it will significantly enhance our operational capabilities.
This is precisely what I was looking for. Is there an estimate for when this will be merged? I don't want to fork the repo just for this.
Hello, thank you for this contribution. I apologize that we've made you wait so long for a proper review. I took a look at the code and it looks really good to me aside from small code duplication in the register targets path; but we can refactor that later. The biggest concern I currently have is that we are storing the external id in plain text. I am engaging with AWS AppSec to understand the best practices around storage of the external id.
Hello, thank you for this contribution. I apologize that we've made you wait so long for a proper review. I took a look at the code and it looks really good to me aside from small code duplication in the register targets path; but we can refactor that later. The biggest concern I currently have is that we are storing the external id in plain text. I am engaging with AWS AppSec to understand the best practices around storage of the external id.
Hi,
AWS IAM documentation state that:
It is not intended to be a 'secret'. Example Corp must provide the ExternalId value to each customer. What is crucial is that it must be generated by Example Corp and not their customers to ensure each external ID is unique.
This external ID is visible to anyone who has IAM read access and access to the AWS account. It is not stored as a secret
Hi! Great job done here. Looking forward to use this feature when approved.
AWS AppSec did not find any problems with this approach. I'm going to engage a teammate to give this another review and then we'll merge it. I'll work on the comments in a follow up PR.
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: marcosdiez, wweiwei-li, zac-nixon
The full list of commands accepted by this bot can be found here.
The pull request process is described here
- ~~OWNERS~~ [wweiwei-li,zac-nixon]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment