Working example of Github OIDC and using Session Tags in IAM policies?

[roskelleycj]

I'm trying to figure out how one would achieve using Github's OIDC w/ AWS AssumeRoleWithWebIdentity.
I can get Github OIDC and aws-actions/configure-aws-credentials to work really well straight out of the box and as advertised. 🎉 (And thank you for making it easy.)

However, when I try to do more complex conditions in IAM policies it seems to fall apart.

By more complex I mean to use Session Tags. E.g., aws:PrincipalTag:Repository which seems to be documented to work. However, this rather cryptic message is also present:

Note that for WebIdentity role assumption, the session tags have to be included in the encoded WebIdentity token. This means that Tags can only be supplied by the OIDC provider and not set during the AssumeRoleWithWebIdentity API call within the Action. 

And in fact when you inspect the code those 'documented session tags' are deleted when using a WebIdentity or webIdentityTokenFile.

And the document seems to indicate that if you need 'Session Tags' you need to look to the Github OIDC for providing those. So how does one do that? How do you provide the expected AWS Session Tag contract in the JWT? When all that you can use to configure Github OIDC is this:

    permissions:
        id-token: write

There does appear to be one other poor soul that has bumped into the same problem.

Is anyone willing and able to educate the uneducated? Thank you for your kindness, in advance. 😊

[roskelleycj]

Correct me if I'm wrong, but most of the examples I found do a double assume role. E.g., the first Assumed Role can only be assumed by the Github OIDC prepared case, and that assumed role only has one permission and that is to allow a second role to be assumed AND at that point aws-actions/configure-aws-credentails can be used again. And it does NOT skip tagging when assuming the second role. And of course this does work!

However, there is nothing preventing anyone inside a github org / repo / branch from cloning the aws-actions/configure-aws-credentials code into their own code base (or similar code), and then using it as a local 'action' and then subverting that code to use a different github org/repo/branch and in essence redirect the security constraints on the second assume role. The rogue code can determine what values to give the session tags at the time the second role is assumed. I know, it is a lot of 'what ifs'. However, in a security conscience environment you don't want any of those what ifs to come to pass, so you ensure that they can not.

That is why Github OIDC w/ the Cloud provider is so great. It establishes w/o question the trusted entity. The issue is that it is weakly defined, thus leading to work arounds that could subvert the security that was intended.

Again, the AWS Session Tags needs to be accomplished w/ the Github OIDC is executed.

For example this invokes the Github OIDC to generate a JWT:

curl -s -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com"

The returned JWT might look like this: (in fact this is exactly what I got back from the above curl, w/ most values changed for security purposes in this thread)

{
  "jti": "<jti>",
  "sub": "repo:<github repo>/<repo name>:ref:refs/heads/<branch name>",
  "aud": "sts.amazonaws.com",
  "ref": "refs/heads/<branch name>,
  "sha": "<sha>",
  "repository": "<github org>/<repo name>",
  "repository_owner": "<github org>",
  "repository_owner_id": "<repository owner id>",
  "run_id": "###",
  "run_number": "##",
  "run_attempt": "#",
  "repository_id": "<repository id>",
  "actor_id": "<actor id>",
  "actor": "<actor name>",
  "workflow": "<workflow name>
  "head_ref": "",
  "base_ref": "",
  "event_name": "push",
  "ref_type": "branch",
  "job_workflow_ref": "<github org>/<repo name>/.github/workflows/<workflow file name>.yml@refs/heads/<branch name>",
  "iss": "https://token.actions.githubusercontent.com/",
  "nbf": ###,
  "exp": ###,
  "iat": ###
}

and ideally, since I specified audience=sts.amazonaws.com when I invoked the Github OIDC it should additionally include the following in the JWT: (look at the very bottom of the JWT example)

{
  "jti": "<jti>",
  "sub": "repo:<github repo>/<repo name>:ref:refs/heads/<branch name>",
  "aud": "sts.amazonaws.com",
  "ref": "refs/heads/<branch name>,
  "sha": "<sha>",
  "repository": "<github org>/<repo name>",
  "repository_owner": "<github org>",
  "repository_owner_id": "<repository owner id>",
  "run_id": "###",
  "run_number": "##",
  "run_attempt": "#",
  "repository_id": "<repository id>",
  "actor_id": "<actor id>",
  "actor": "<actor name>",
  "workflow": "<workflow name>
  "head_ref": "",
  "base_ref": "",
  "event_name": "push",
  "ref_type": "branch",
  "job_workflow_ref": "<github org>/<repo name>/.github/workflows/<workflow file name>.yml@refs/heads/<branch name>",
  "iss": "https://token.actions.githubusercontent.com/",
  "nbf": ###,
  "exp": ###,
  "iat": ###,

  "https://aws.amazon.com/tags": {
        "principal_tags": {
            "Repository": ["<GITHUB_REPOSITORY>"],
            "Actor": ["<GITHUB_ACTOR>"],
            "Branch": ["<GITHUB_REF>"],
            "Commit": ["<GITHUB_SHA>"],
            "Workflow": ["<GITHUB_WORKFLOW>"]
        }
    }
}

or something similar, that meets the contract for AWS IAM roles to establish BOTH the trust for the role, as well as to utilize the session tags in IAM policies associated w/ the role. This would prevent having two roles AND would allow one to create an AWS IAM role that is both assumable by Github actions AND contains exact information in the session tags to allow complex policies to be created and governed by the values of the aws:PrincipalTag. E.g., I can have policies that are specific to a github org, or repo or branch or workflow. This really provides very powerful constructs to allow the ease of workflows in Github and utilizing AWS Resources for CI/CD.

🤷 Again, I'm probably uneducated and need more education to understand what the expected approach is.

[manics]

I'm not sure if this helps you, but the example in the README works fine if you add a Policies section to the role with the permissions. Are you needing the second assume-role to achieve something else?

[roskelleycj]

@manics, yes I've accomplished that part. But if you notice there are no 'policies'. They are missing. That is the part that I want to add. And more importantly I want to add them to the role that can be assumed by aws-actions/configure-aws-credentials. Not a two role combination.

However, because the Github OIDC is not implemented correctly (as to my understanding ) then you can not make policies in that role w/o making them very, very specific to either the github org or the repo/etc. E.g., I have 1600+ repos spread over a number of github organizations. Meaning I would have a large number of roles.

The ideal is what AWS describes in that if the following (or something like it) is provided by Github OIDC:

"https://aws.amazon.com/tags": {
        "principal_tags": {
            "Repository": ["<GITHUB_REPOSITORY>"],
            "Actor": ["<GITHUB_ACTOR>"],
            "Branch": ["<GITHUB_REF>"],
            "Commit": ["<GITHUB_SHA>"],
            "Workflow": ["<GITHUB_WORKFLOW>"]
        }
    }

then a very few roles can be defined AND have very specific policies that include the Github provided information. Information that can not be spoofed.

For example. Suppose I have an S3 bucket that we use for 'dynamic configuration'. And further suppose that Github OIDC provided the AWS Session Tags as described. Then I could add the following policy statement to the Role:

{
  "Action": [
    "s3:GetObject",
    "s3:DeleteObject",
    "s3:PutObject"
  ],
  "Effect": "Allow",
  "Resource": [
    "<some bucket arn>/env/prod/repo/${aws:PrincipalTag/Repository}/private/*",
    "<some bucket arn>/env/prod/repo/${aws:PrincipalTag/Repository}/shared/*",
  ]
}

If you'll notice, I'm able to use the <GITHUB_REPOSITORY> from the AWS Session Tag in the policy. This means that when I create a github workflow that uses Github OIDC along w/ aws-actions/configure-aws-credentials I have a single role for all 1600+ repos in all the github organizations that I trust w/ this role to copy files from the content in the repo into the S3 bucket. E.g., I don't have to have lots of roles per repo / github org combination.

This is the power of Github OIDC and AWS, when implemented correctly.

[roskelleycj]

Another thought. If when the Github OIDC is generated the '<GITHUB_ACTION>' could be identified and added to the AWS Session Tags, then this would allow another layer of trust. E.g., I'm assuming a lot about actions. In that if actions are sourced from github repos, and all of the security that github org/repos imply, then I have one additional layer of trust that can be established. In that I can make roles that only trust aws-actions/configure-aws-credentials. Or if I really want to go to the work of writing my own action. I can place that action in a github org/repo that only 3 people have access to, thus limiting the possible attack vectors.

Again, this too would be quite powerful in the Github OIDC and AWS combination.

[jasondewitt]

I will chime in with my use case, because I am facing similar issues.

I have a working OIDC connection to a single role in one of my AWS accounts, but I have over 100 accounts to set up for OIDC. This connection works and access can be controlled by allowing each repo in the IAM role trust policy. I was planning to create an OIDC provider and role in each account, but quickly found that this will not easily scale.

I would like to use a single account in our network as the entry point for OIDC connections, and then that session will assume another role in the target account. This is currently documented and working, however, in these secondary assumed roles, in the target aws accounts, I want to be able to have a condition which checks which repo the request is coming from in the IAM trust policy.

It appears this should be possible with Session Tagging, but like @roskelleycj has been demonstrating, no session tags are being set on the OIDC session. The documentation for this action appears to indicate that that Session Tags are supported for OIDC sessions, but it is unclear if that is actually the case:

The action will use session tagging by default during role assumption. Note that for WebIdentity role assumption, the session tags have to be included in the encoded WebIdentity token. This means that Tags can only be supplied by the OIDC provider and not set during the AssumeRoleWithWebIdentity API call within the Action.

Is this a deficency in GitHub's JWT implementation? or something to fix in this action? Its starting to look like this is required on the GitHub side and is not supported...

[roskelleycj]

Looks like @glassechidna ran into a similar issue and made a novel workaround. Although, his approach to the problem is different than mine. In that he wants AWS to change their STS implementation. 🤷 I figured since AWS documented what their approach AND Github OIDC shows that you must provide the audience=sts.amazonaws.com that the two tunnels would align in the middle somehow. 😊

[roddyherries]

Hi @roskelleycj, I am facing the same challenge. If these custom claims were included in the JWT generated by Github OIDC it would be the missing piece to writing powerful and secure IAM policies. I am now faced with the choice between permissive single policies shared by many GH repos or creating specific policies for each repo. Have you had any traction with Github on this?

[RichiCoder1]

Unfortunately the fact that the tags must be in the JWT makes this a GitHub issue more than an AWS issue. AFAIK, the only thing that GitHub lets consumers modify about the JWT is the Audience. Otherwise it's entirely immutable. So either GitHub would need to allow appending extra claims or AWS would need to add session tag mapping to OIDC providers.

[peterwoodworth]

You nailed it @RichiCoder1, this needs to be defined in the JWT because AssumeRoleWithWebIdentity() lacks a tags parameter like AssumeRole() does. Meaning AWS doesn't provide session tag mapping for OIDC directly.

However, I'm unsure how or if there is a way to modify the JWT from GitHub in the necessary fashion. If anyone knows a way to do this, please leave a comment 🙂

[JTarasovic]

Does this announcement remove the service limitation? https://github.blog/changelog/2022-10-18-github-actionsopenid-connect-support-enhanced-to-enable-secure-cloud-deployments-at-scale/

[RichiCoder1]

Unfortunately no. That just allows customizing the subject claim, while this ticket requires adding additional custom claims.

[ScottGuymer]

I too am running into the same issue.

I want to be able to write an OIDC policy that will allow each repo to write to its own folder in a bucket. We have over 8k repos so not really an option to have a role for each repo.

The claims are there in the JWT, but AWS doesnt let you access them directly The spec for tags is there from AWS, just GH doesnt implement it..

This could be fixed from either end really.

Being able to use the claims or the tags would be very powerful for a lot of things.

Is anyone aware if GH are working on this?

[roskelleycj]

As mentioned this 'novel hack' appears to continue to work well, with several key modifications to the Lambda Function to limit the blast radius. For example the job_workflow_ref claim can be used to control what org/repo/.github/workflow/.yaml has access to the IAM Role. Additionally, this works well as you can put all of this in a central AWS Account using APIGateway and you can do STS Assume Role to the various accounts that need to be accessed.

It would be much simpler if either GitHub or AWS or BOTH worked together to find a solution to allows the AssumeRoleWithWebIdentity to contain tags based upon the JWT. Even if the only tag was the sub with the customization of the OIDC this alone would give us enough control to make something customizable AND secure.

One note about the novel hack, sometimes APIGateway and GitHub Actions do not agree about the time that the JWT was issued. (E.g., it looks like GitHub's clocks are in the future occasionally) and AWS APIGateway rejects the request out of hand. 😞 a jitter has to be applied OR a few retries have to be applied, thus making it slower too.

[aidansteele]

@roskelleycj I am the author of the novel hack. I wrote it in a rush shortly before GHA officially launched OIDC support, assuming (hoping) that better integration would arrive sooner rather than later. It seems that wish hasn't come true yet. I didn't realise anyone was actually using it.

A few months later (the start of this year) I wrote a follow up solution that I've since been using. I built it on the assumption that a) a more general solution might be more useful and b) it might be preferable to still use OIDC providers in AWS IAM. I originally wanted to include every claim in Github's JWT, but role session tags are limited to 500 bytes 😢

~~For better or worse, AWS take feature requests filed through tech support tickets most seriously~~ (See correction below). I wrote some requests in a blog post about how I'd prefer it work, but I feel that we're in the minority caring about it this much.

[lorengordon]

@aidansteele I would also like to see that feature! I find it so exhausting opening tickets with tech support for feature requests. Same response every time, then feels like it goes into a black hole. No visibility, no updates.

[kellertk]

For better or worse, AWS take feature requests filed through tech support tickets most seriously.

That's not necessarily true: when you file a request with the Technical Support button in the console, you're sending that request over to the AWS Premium Support team. Premium Support will then send that ticket over to the responsible AWS service team as a feature request. Likewise, when you file a feature request that's applicable to a service in GitHub, our team will create a ticket and send it to the same AWS service team. Many teams at AWS do prioritize feature development based on which customers need it, but the requests end up in the same bucket for evaluation most of the time. It is very true that we use demand to prioritize work though. Number of customers that submit a ticket through AWS PS is one data point that teams use to evaluate priority, but another one is the number of people that have a +1 reaction on a GitHub issue (you can take a look at some of the feature requests in the AWS SDKs on GitHub and see all the +1s we have), so we encourage people to use GitHub reactions as well :)

Looking at the blog post by @aidansteele, it looks like you're asking for two different behavior changes service side: IAM role trust policies should have a permissions boundary, and a way to map arbitrary claims from the OIDC token to role session tags. Cool! We'll forward those over to the correct teams.

There's still a little bit of research that the team that develops this action needs to do around this subject, so I'm leaving this open while we do that, but I wanted to let @aidansteele know that we read the blog post and generated some action items from it for the responsible teams.

[aidansteele]

@kellertk Thank you so much for the detailed correction. It's really great to hear that these requests aren't "disappearing into the void". I'll be sure to 👍 more feature requests when I see them on Github, I didn't realise they carried so much weight.

Specific to this topic: it's also great to hear that the service teams are aware of the requests. I can only imagine how much work is involved in making fundamental changes like that to such a sensitive system.

[dcopso]

@kellertk Until the tagging issues above are addressed, it would be great to get the README updated because it is currently very misleading. Specifically, https://github.com/aws-actions/configure-aws-credentials#session-tagging should clarify that tags are not usable when the action is configured for AssumeRoleWithWebIdentity. Thank you.

[icaliskanoglu]

I have same issue. We have ~100 github repositories that push docker images to ecr. I did not want to create roles for every repo. I worked around with 2 roles and assume chain. First role: Get credentials with assumeRoleWithWeb identity, second role to tag session with repo name. With this chain I am able to use aws.PrincipalTag/Repository in the condition.

 - name: configure aws credentials
    uses: aws-actions/configure-aws-credentials@v2
    with:
      role-to-assume: arn:aws:iam::${{ vars.ACCOUNT_ID }}:role/GenericGHActionRole
      role-session-name: TempSession
      aws-region: us-west-2
  - name: configure aws credentials
    uses: aws-actions/configure-aws-credentials@v2
    with:
      role-to-assume: arn:aws:iam::${{ vars.ACCOUNT_ID }}:role/ImageBuilderRole
      role-session-name: ImageBuilder
      aws-region:  us-west-2
      role-chaining: true
  - name: Login to Amazon ECR
    id: login-ecr
    uses: aws-actions/amazon-ecr-login@v1

[dcopso]

@icaliskanoglu Thanks for sharing that. Would you be willing to share the policies/roles you are using for the two-step role chaining?

[ajmilazzo]

@icaliskanoglu What is the role trust policy that you have for the first and second role? As well as the permission policy for the first role? From my understanding, you need to allow sts:TagSession to pass tags when you assume the second role. However, it is not clear how you would restrict this.

If sts:TagSession is unrestricted, and the permission policy in the second role trusts the PrincipalTag, it might allow a malicious actor to call sts directly and inject whatever tags they want into the second session.

[kedare]

So there are no way to achieve this without having to do a double assume role ? Is there any reason why it's not possible ? That make things much more complicated, we would have to define 2 roles for each use cases ?

Is there a way to have a generic first role to use everywhere and then the on the second assumerole to filter depending on which repository it ran by, to which final role it can go ?

[vixus0]

Is there a way to have a generic first role to use everywhere and then the on the second assumerole to filter depending on which repository it ran by, to which final role it can go?

@kedare Yes, add a condition on the trust policy of the second role.

For example, if you have a generic first role called generic-gh-role that can be assumed by any repository in your org, then the trust policy on your second role would be:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/generic-gh-role"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalTag/Repository": "my-org/my-repo"
        }
      }
    }
  ]
}

But this doesn't avoid the security issue mentioned by @roskelleycj in https://github.com/aws-actions/configure-aws-credentials/issues/419#issuecomment-1102065795 where a malicious actor could use a fork of this action to override these tags.

The only proper fix is:

• GitHub adds the "https://aws.amazon.com/tags": ... to their JWT (unlikely)
• OR AWS allows mapping JWT claims to IAM session tags (as described already by @aidansteele).

[unlobito]

@kedare There's an additional solution that's not been mentioned so far: After https://github.com/aws-actions/configure-aws-credentials/pull/739, you can declare an inline-session-policy alongside your call to AssumeRoleWithWebIdentity

We use this to define a Reusable Workflow that declares an inline-session-policy restricting access to AWS resources based on the calling repository (using github.repository from the github context to determine the calling repository at runtime).

Next, we use a customised sub OIDC claim (see also, Customizing the token claims) to enforce job_workflow_ref matches our trusted Reusable Workflow, preventing the inline-session-policy from being manipulated by workflows not managed by our infrastructure team.

Finally, we leverage "Approving workflow runs from private forks" to ensure the sub claim for job_workflow_ref can't be confused through unexpected PRs against our Reusable Workflow.

--

We've found this approach to be "as close as we can get" to an IAM native solution, while still retaining clear escape paths once either aws-actions/configure-aws-credentials#419 (this issue!) or community/discussions#12031 are resolved.

We've internally +1'd this issue with our AWS and GitHub representatives, but would encourage interested community members to do the same.

Hope that helps 😃

[jetersen]

@unlobito would be great with an example for all the steps. Cause I have a hard time piecing the docs together without a concrete example of how to achieve within a GitHub Reusable workflow.

Perhaps share an example of identity provider policy json document and an example workflow.

I was baffled that reusable workflow OIDC is not usable together with AWS STS 😓 I have Google workload identity working with reusable workflows no problem...

[vixus0]

@unlobito that's a cool idea! What policy is required for the role being assumed? Does it act as a sort of permissions boundary for the inline-session-policy?

[unlobito]

@vixus0 That's exactly right- the IAM policy being assumed mostly has wildcard permissions across resources on the account, but the IAM role's policy document enforces our aud matches GitHub and sub matches our Reusable Workflow (so we expect the wildcard privileges to never be directly assumable, assuming the Reusable Workflow remains unmodified).

Using wildcards is non-ideal, but we expect we can get rid of the inline-session-policy and the role wildcards once IAM Session Tags supports reading in from OIDC JWT claims.

@jeterson I'm still on annual leave for a few more days, but hopefully the info above is useful in the meantime.

I'll try to get approval for releasing samples of our deployed IAM policy + Reusable Workflow once I'm back at work though 😊

[unlobito]

Hey @vixus0 @jeterson,

As promised, we've just released a sample of our setup for enforcing per-repo privileges through inline-session-policy in a Reusable Workflow 🎉

https://github.com/Skyscanner/gha-aws-oidc-sample

Unfortunately, this doesn't fully solve the issue we're currently on (since session tags remain unavailable until either GitHub or AWS update their OIDC implementations), but does deliver on identifying individual repositories in CloudTrail through role-session-name.

I'd still recommend +1ing this issue with your AWS and GitHub representatives, but hope this is useful as a workaround in the meantime :)

---

This wouldn't be complete without a massive "Thank You" to my Skyscanner colleagues for help with design and review- shout outs to my $InfrastructureTeam colleagues (especially @dimitar-hristov + @michaelroddy's work on https://github.com/aws-actions/configure-aws-credentials/pull/739), our colleagues in $SecurityTeam, and our internal Open Source group.

…and of course, thank you (the community!) for all of your notes on this issue + community/discussions#12031 💚

[djonser]

This is all doable using Cognito Identity as has been mentioned in a separate issue, but it 's not something that has been well documented/tested. I spent some time to do such and decided to create my first open source contribution related to this.

Maybe some of you can find something that helps you out over at https://github.com/catnekaise and help test/review this. Details of how we would use Cognito Identity in GitHub Actions to enable ABAC can be found here.

[stekern]

Great and thorough work, @djonser 👏 I wasn't aware of the possibility of using Cognito for this. I've taken a quick look at your repositories and website, and I'll be trying this out! Thank you for sharing code, examples and documentation with the community - very much appreciated.