Support Role profile with SSO profile as a source profile.

[MarcinKasprowicz]

Is your feature request related to a problem? Please describe. The following scenario isn't covered by the plugin

• Role profile (account B) with SSO profile (account A) as a source profile.

# ~/.aws/config file

[profile payment-suite-pro]
sso_start_url = https://xxx.awsapps.com/start
sso_region = eu-north-1
sso_account_id = 99999999999
sso_role_name = FullAccess

[profile spt-payment-dev]
role_arn = arn:aws:iam::111111111:role/AssumableAdmin
source_profile = payment-suite-pro

Current behaviour

➜ npx sls info

> [email protected] deploy:info:order-service
> sls info

Environment: darwin, node 16.17.0, framework 3.22.0 (local), plugin 6.2.2, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Cannot resolve serverless.yml: Variables resolution errored with:
  - Cannot resolve variable at "provider.iam.role.statements.1.Resource.0": Profile spt-payment-dev did not include credential process

Describe the solution you'd like

$ export AWS_PROFILE="spt-payment-dev"
$ npx sls info
# 1. Initiate aws sso login towards 'payment-suite-pro' account
# 2. Get temporary credentials using AssumeRole action to act as 'spt-payment-dev' account

[thomasmichaelwallace]

Hi @MarcinKasprowicz -

Thanks for taking the time to fill in the issue and write a pull request.

I'm a little bit unsure of the use case? As far as I can see this is quite a non-standard way of authenticating (I'm not sure the aws-cli or aws-sdk would support this config?).

The built-in AWS SSO already works with organisations to allow cross account access with a single SSO sign-in url; which is what I use to authenticate across multiple accounts. This has the added advantage that the credentials can be refreshed directly if you do something which outlasts the sso/role assumption timeouts.

For example:

[profile payment-suite-pro]
sso_start_url = https://xxx.awsapps.com/start
sso_region = eu-north-1
sso_account_id = 99999999999
sso_role_name = FullAccess

[profile spt-payment-dev]
sso_start_url = https://xxx.awsapps.com/start
sso_region = eu-north-1
sso_account_id = 111111111
sso_role_name = AssumableAdmin

There's some more information in this documentation section: https://docs.aws.amazon.com/singlesignon/latest/userguide/manage-your-accounts.html

I'd love to understand the reason you're using this approach of manually assuming the role (instead of using SSO + organisations) and whether it's something you've found supported elsewhere before taking on this additional feature.

[MarcinKasprowicz]

Hi @thomasmichaelwallace

Thanks for asking those questions, I could take a look one more time at our approach and validate it 🧐.

I realized that we could try to update the trust relationship of AssumableAdmin role so that it could support the configuration that you have shown in the example however ... SSO is governed by a central team and I'm not sure if that would be possible without their intervention. 🤷‍♂️

In our setup, I wanted to have one entry point to our accounts. That entry point is the bastion account. To act in the context of a given child account we assume a role (AssumableAdmin). Anything from payment-suite-pro can assume it. IAM user as well SSO federated user.

This flow works brilliantly with Terraform (and assume role capability https://learn.hashicorp.com/tutorials/terraform/aws-assumerole)

# on local machine
# auth as SSO federated user from payment-suite-pro
➜ asp payment-suite-pro
➜ aws sso login

➜ cd infrastructure/environments/spt-payment-dev
# Terraform will automatically assume role `AssumableAdmin` of spt-payment-dev
➜ terraform state list
➜ terraform apply

# Terraform will automatically assume role `AssumableAdmin` of spt-payment-stg
➜ cd ../../../infrastructure/environments/spt-payment-stg
➜ terraform state list
➜ terraform apply


# in ci/cd
# auth as IAM user from payment-suite-pro
➜ export AWS_ACCESS_KEY_ID=<id>
➜ export AWS_SECRET_ACCESS_KEY=<secret>

➜ cd infrastructure/environments/spt-payment-dev
# Terraform will automatically assume role `AssumableAdmin` of spt-payment-dev
➜ terraform apply

➜ cd ../../../infrastructure/environments/spt-payment-stg
# Terraform will automatically assume role `AssumableAdmin` of spt-payment-stg
➜ terraform apply

There is also one subtle thing that is pretty nice. You can switch between accounts using Role history by less clicks than with AWS SSO.

With the current state of Serverless (without the plugin) we need to do something like

# on local machine
# auth as SSO federated user from payment-suite-pro
➜ asp payment-suite-pro
➜ aws sso login

➜ ./execute_some_assume_role_script.sh arn:aws:iam::111111111:role/AssumableAdmin 
➜ sls invoke ...
➜ sls info --stage dev
➜ sls deploy --stage dev

➜ ./execute_some_assume_role_script.sh arn:aws:iam::22222222:role/AssumableAdmin
➜ sls invoke ...
➜ sls info --stage stg
➜ sls deploy --stage stg

# in ci/cd
# auth as IAM user from payment-suite-pro
➜ export AWS_ACCESS_KEY_ID=<id>
➜ export AWS_SECRET_ACCESS_KEY=<secret>

➜ ./execute_some_assume_role_script.sh arn:aws:iam::111111111:role/AssumableAdmin
➜ sls deploy --stage dev

➜ ./execute_some_assume_role_script.sh arn:aws:iam::22222222:role/AssumableAdmin
➜ sls deploy --stage stg

With the change that I proposed:

# on local machine
➜ asp spt-payment-dev
➜ sls invoke ...
➜ sls info --stage dev
➜ sls deploy --stage dev

➜ asp spt-payment-stg
➜ sls invoke ...
➜ sls info --stage stg
➜ sls deploy --stage stg

# in ci/cd
# auth as IAM user from payment-suite-pro
➜ export AWS_ACCESS_KEY_ID=<id>
➜ export AWS_SECRET_ACCESS_KEY=<secret>

# In case if I would be just deploying in ci/cd
# I could use that approach - https://www.serverless.com/framework/docs/providers/aws/guide/credentials#assuming-a-role-when-deploying. 
➜ sls deploy --stage dev
➜ sls deploy --stage stg

I think that I would be able to solve my problem by updating the trust relationship in AssumableAdmin... I will try to do it.

Does aws-cli or aws-sdk support the flow that I have mentioned? Yes it does.

➜  ~ asp spt-payment-dev
➜  ~ aws sts get-caller-identity | jq
{
  "UserId": "a",
  "Account": "1111",
  "Arn": "arn:aws:sts::1111:assumed-role/AssumableAdmin/botocore-session-1664355499"
}
➜  ~ asp spt-payment-stg
➜  ~ aws sts get-caller-identity | jq
{
  "UserId": "b",
  "Account": "22222",
  "Arn": "arn:aws:sts::2222:assumed-role/AssumableAdmin/botocore-session-1664355512"
}

[thomasmichaelwallace]

Sorry it’s taking awhile to get back to you on this.

I’m reluctant to bake in a company-specific authentication flow. Especially one that is a custom version of what AWS supports natively. So I want to do a bit of testing to see how the aws-sdk-js is handling this itself.

The plugin should be using all the same ini file source profile resolution that the aws-sdk does, and so, if everything is just working natively for you, then that might be the heart of this bug.

It is surprising how many workarounds have their origins in ‘because central IT…’.. At a glance, at least, it does seem like you’re rolling your own version of the the AWS IAM Identity Centre + AWS Organisations feature set; a centralised login that gives you access, via controlled role assumption, into organisation accounts.

I don’t know if you appreciated that the asp isn't actually part of the aws-cli or aws-sdk, it's part of zsh’s AWS plugin (https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/aws#plugin-commands). I think this might turn out to be important because asp [profile] login seems to resolves SSO credentials to static ones, which would explain some differences in behaviour.

Last resort - AWS does provide an escape hatch for people using company-specific approaches with the credentials_process configuration (We used the credential process approach to support SSO before AWS supported it natively). The complete documentation is here: https://docs.aws.amazon.com/sdkref/latest/guide/feature-process-credentials.html - but you can use acp (from the zsh plugin) followed by the aws cli to output whatever combination of SSO + role assumption you wanted.

[RealityCtrl]

Adding a +1 on this request as we've hit the same issue while we are migrating to SSO (with what looks like exactly the same approach). So we would ideally like the functionality as outlined in this enhancement to be able to seamlessly work with serverless and SSO.

[thomasmichaelwallace]

That's good to know @RealityCtrl .

I'd love to know (for my own interest) why you're not using AWS' IAM Identity Centre to solve this problem and, instead, rolling your own with bastion + roles? Central IT again?

Nevertheless, it does prove out that people are going to do this.

I actually think the fix is not a separate, specialist, flow (as currently implemented in the PR), but to fix the bug that the SharedIniFileCredentials is not respecting source profiles which use the SSO implementation. So I'll take a look at that when I can.

[RealityCtrl]

It's down to a centralised cloud engineering team that owns the setup of commonalities like AWS accounts, VPCs etc. There is probably the historical context of how the AWS accounts were setup in the company when AWS was fairly new. All I can say is right now there is a user that can log into a base account and then jump to a role in the same of other accounts. It seems like this is being replicated somewhat with single sign on. I don't have any context behind the decision making beyond that or know much about AWS IAM Identity Center myself.

[erikerikson]

Hey @thomasmichaelwallace Thank you very much for this project. It got me off the ground. FWIW I am also running into this limitation and I'm woring to get deeper.

TL;DR: the v3@aws-sdk iniFile loader looks like it should work (for this and credential_source)(as opposed to the currently used v2).

Use Case Multiple account structure where developers SSO into each account (assuming a role therein) via IAM Identity Center with an external IdP. Both developers and the CICD pipeline which has credentials from my root account assume a cicd deployment role and use it to deploy (developers usually only do this to ensure parity with the build system). FWIW, this does work with the AWS CLI. FWIW, there are a subset of workflows where deployment spans accounts (applies to all of a subset of them) and thus a multi-profile setup is used by different commands at different phases of the CICD process.

admin's ~/.aws/config:

[profile sand]
sso_start_url = $START_URL
sso_region = $REGION
sso_account_id = $SAND_ACCOUNT_ID
sso_role_name = $ROLE_NAME
region = $REGION
output = json

[profile sand-cicd]
role_arn = arn:aws:iam::$SAND_ACCOUNT_ID:role/cicd
source_profile = sand

[profile prod]
sso_start_url = $START_URL
sso_region = $REGION
sso_account_id = $PROD_ACCOUNT_ID
sso_role_name = $ROLE_NAME
region = $REGION
output = json

[profile prod-cicd]
role_arn = arn:aws:iam::$PROD_ACCOUNT_ID:role/cicd
source_profile = prod

CICD:

[profile sand-cicd]
role_arn = arn:aws:iam::$SAND_ACCOUNT_ID:role:role/cicd
credential_source = Environment

[profile prod-cicd]
role_arn = arn:aws:iam::$PROD_ACCOUNT_ID:role:role/cicd
credential_source = Environment

Do you have an issue ref for the SharedIniFileCredentials bug or are you just noting that there seems to be a bug in the aws-sdk class? A quickish look into that sdk code and repo didn't turn up an issue and seems to confirm that SSO credentials are ignored by the ini file loader (also credential_source). After dipping my toe into this, I wonder if the solution might be an sdk 2 -> 3 update and switching to using similar logic to the v3 @aws-sdk's fromnodeproviderchain (code)which looks like the auto-magical load credentials from wherever method to which parity(ish) and the Serverless specific embellishment should be added.

[thomasmichaelwallace]

Hey @erikerikson !

So here's my take:

• credential_source isn't supported by the aws-sdk js v2, and won't be because (especially with the release of Node 18 lambdas bundled with v3), AWS do not intend to support v2 much longer: https://github.com/aws/aws-sdk-js/issues/1916
• although it looks like someone did backport the new [sso-session my-sso] token provider approach (aside- your config file is now using the legacy(!) "sso" configuration.)
• source_profile explicitly says that the "[source_profile] chain is stopped when the SDK encounters a profile with static credentials". SSO credentials are not static, which is why the aws-sdk js v2 doesn't support it.
• however boto (and therefore the aws-cli) recursively resolves source_profiles like each profile is completely independent, which is why everyone enjoys this incorrect (?) or, at the very least, poorly defined behaviour in the CLI; and if everyone is using it then it is, at least, de facto correct now 😄
• the challenge for supporting it with v2 is that the assumption that a source_profile is static is hardcoded: https://github.com/aws/aws-sdk-js/blob/master/lib/credentials/shared_ini_file_credentials.js#L234 into the provider, so you have to write your own role resolver
• however the AWS did fix this in v3 https://github.com/aws/aws-sdk-js-v3/blob/main/packages/credential-provider-ini/src/resolveProfileData.ts (as well as providing support for the new SSO format and credential_source)
• but v3 is a fairly significant departure, and the serverless framework uses aws-sdk v2, which means that we need a bridge to convert the AWS credentials interface between v3 and v2 if we use v3 credentials!
• note that it is a mistake to just provide the literal credentials (key, secret and/or token) to an aws service config (like the serverless framework currently does!) because most credentials are not synchronous and also require refreshing; which is something this plugin fixes
• the company I work for (and wrote this for, and the patch that predated it) actually gave up on the serverless framework - I would strongly recommend that people consider the AWS CDK now - which side steps pretty much all the credential problems; but also means I have less time to fill in gaps.

So... where does that leave us?

Probably the solution is:

1. Write a bridge between v2 and v3 credentials
2. Re-implement the SSO provider using v3 (so that it can do the prompting)
3. Re-implement the chain provider using v3 (so that serverless' existing bespoke credential resolution order is preserved)

But it's probably not something that's going to get done in the short term.

[erikerikson]

Oh man, I didn't mean to ask for such a reply. Thank you very much, its incredibly useful.

Oddly, that config comes from a very recent "latest" awscli install.

Thank you for the CDK recommendation. I've been previously warned off of it but I've also hit a fairly large set of barriers in the framework. I would definitely miss the variable system... /hrmm

Above these, thank you so much for the analysis as well as the clear understanding of your incentives. Sounds like I have some empowered decision making to do next :D

Thank you for your kindness and all the best!

[thomasmichaelwallace]

No worries -

It was helpful getting everything down in a (semi) structured way; and give everyone waiting on this issue a bit more context 😄

[deathemperor]

pardon for my confusion but does this plugin works with AWS JS SDK v3 (either on node 16 or 18 runtime)? I've been trying to migrate to aws js sdk v3 with no luck

[thomasmichaelwallace]

@deathemperor this plugin isn't written using v3.

However it is fine for the version your tooling (i.e. serverless framework) is using and the version you lambda functions are using to be different.

You can install v2 and v3 side-by-side (although I expect the serverless framework would have installed v2 automatically):

npm install --dev aws-sdk # now your tooling can use v2
npm install @aws-sdk/client-s3 # now your code can use v3

[deathemperor]

@thomasmichaelwallace all my tooling works just fine, it's just that when deployed using any sdk module throws CredentialsProviderError: Could not load credentials from any providers

[thomasmichaelwallace]

@deathemperor - I've moved this discussion to a new issue #20 - as it is unrelated to SSO - let's continue to discussion there.

[navrkald]

Please any update about this issue?

It's been one and half year until creation. Our company is forcing us to use this approach as well and we had to invent crazy workarounds on our side to deal with sso login using one role and then switching to another role. When using AWS CLI, this approach is natively supported and when reading comments above, seems like this approach is very popular in corporates. Solving this issue on plugin side to behave like native AWS CLI will solve our pain. Thanks for understanding.

[RealityCtrl]

Serverless will support this in V4 it seems https://github.com/serverless/serverless/issues/7567#issuecomment-1912912877