AWS_PROFILE not properly used when verifying with AWS KMS

[joelapatatechaude]

trafficstars

I am currently focusing on cosign verify, but what I am writing probably applies to cosign sign too. I am using AWS KMS to sign images in ECR.

There are several things that I find strange when using AWS_PROFILE. For one, even if my .aws/config specify a default aws region for all my profiles, it seems like I always need to specify the env variable AWS_REGION when using cosign, if not, then I get an error asking me for a region. It's not that important, but it's a sign there is maybe something wrong in the implementation.

More importantly: If I run from an ec2 instance something like:

AWS_PROFILE=ch AWS_REGION=ap-southeast-2 cosign verify -key awskms:///alias/my_key_alias  6xxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my_repo/my_image:latest

I get:

error: no matching signatures: getting public key: AccessDeniedException: User: arn:aws:sts::6xxxxxxxxxxx:assumed-role/MyInstanceRole/i-0a94xxxxxxxxxx is not authorized to perform: kms:GetPublicKey on resource: arn:aws:kms:ap-southeast-2:6xxxxxxxxxxx:key/my_key_id

Why is cosign trying to use my instance profile role, instead of the role I provide in AWS_PROFILE?

The profile I specify in AWS_PROFILE I am using can get the public key, as the following command succeeds

AWS_PROFILE=ch aws kms get-public-key --key-id arn:aws:kms:ap-southeast-2:6xxxxxxxxxxx:key/my_key_id

Now, if I give my ec2 instance profile permission to use the key, then cosign will succesfully verify when I specify AWS_PROFILE=ch (which should force using the ch profile, not the ec2 instance profile). However, if I try to verify without specifying an AWS_PROFILE at all, then cosign can't verify.

Overall, the behavior of cosign with AWS KMS using AWS_PROFILE seems strange.

[cpanato]

The way we are loading the aws credentials we expect the keys are available and not the profile.

I see this as a missing feature and I will fix that

[cpanato]

ok, using my local machine everything works as expected, will try your use case

[joelapatatechaude]

Ok. Not sure the way I am using aws profile is part of the problem, but it's not a basic profile usage, as the profile ch is assuming a role (role_arn) using the creds from another profile (source_profile). AWS Virutal MFA is enabled, (aws_mfa_device in the credentials file). To simplify the usage, the aws_access_key_id, aws_secret_access_key, aws_session_token, and aws_security_token are automatically generated via aws-mfa tool.

[cpanato]

ok, seems not a bug and everything is working as expected

ubuntu@ip-172-19-10-136:~$ cosign verify -key awskms:///alias/test-carlos  ctadeu/test:123
error: no matching signatures:
getting public key: MissingRegion: could not find region configuration


ubuntu@ip-172-19-10-136:~$ export AWS_REGION=us-east-1
ubuntu@ip-172-19-10-136:~$ cosign verify -key awskms:///alias/test-carlos  ctadeu/test:123
error: no matching signatures:
getting public key: AccessDeniedException: User: arn:aws:sts::XXXXX:assumed-role/XXXXXX/i-0XXXXXa is not authorized to perform: kms:GetPublicKey on resource: arn:aws:kms:us-east-1:XXXXXX:key/c2XXXXf-XXXXX
        status code: 400, request id: baXXXX-XXX-XXX-XXX-7XXXXXc4


ubuntu@ip-172-19-10-136:~$ mkdir -p .aws
ubuntu@ip-172-19-10-136:~$ nano .aws/credentials
ubuntu@ip-172-19-10-136:~$ export AWS_PROFILE=test
ubuntu@ip-172-19-10-136:~$ cosign verify -key awskms:///alias/test-carlos  ctadeu/test:123

Verification for ctadeu/test:123 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":"index.docker.io/ctadeu/test"},"image":{"docker-manifest-digest":"sha256:6fe11397c34b973f3c957f0da22b09b7f11a4802e1db47aef54c29e2813cc125"},"type":"cosign container image signature"},"optional":null}]



ubuntu@ip-172-19-10-136:~$ unset AWS_PROFILE
ubuntu@ip-172-19-10-136:~$ unset  AWS_REGION
ubuntu@ip-172-19-10-136:~$ cosign verify -key awskms:///alias/test-carlos  ctadeu/test:123
error: no matching signatures:
getting public key: MissingRegion: could not find region configuration

ubuntu@ip-172-19-10-136:~$ AWS_REGION=us-east-1 cosign verify -key awskms:///alias/test-carlos  ctadeu/test:123
error: no matching signatures:
getting public key: AccessDeniedException: User: arn:aws:sts::XXXX:assumed-role/XXXX/i-0af7f137020626ada is not authorized to perform: kms:GetPublicKey on resource: arn:aws:kms:us-east-1:XXXX:key/XXXXX-XXX-XXX-XXXX-XXXX
        status code: 400, request id: XXXXX-12ea-XXXX-XX-XXXXX


ubuntu@ip-172-19-10-136:~$ AWS_PROFILE=test AWS_REGION=us-east-1 cosign verify -key awskms:///alias/test-carlos  ctadeu/test:123

Verification for ctadeu/test:123 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":"index.docker.io/ctadeu/test"},"image":{"docker-manifest-digest":"sha256:6fe11397c34b973f3c957f0da22b09b7f11a4802e1db47aef54c29e2813cc125"},"type":"cosign container image signature"},"optional":null}]

[cpanato]

you need to export the AWS_REGION the config file looks like it only works with the aws CLI, I see similar behavior when using terraform and aws profile.

[cpanato]

humm I see, yep that looks like very special case

[joelapatatechaude]

Hi @cpanato Thanks for having looked into it. I'll see if I can narrow down my issue to something easily reproducible.

[joelapatatechaude]

Hi @cpanato, I am back :o) I still feel there is a problem here. I am runnig cosign 1.6.0 I have 2 aws account: AAA from where I sign the image (ec2 in AAA), and BBB where the kms and ecr repo live.

AWS_REGION=ap-southeast-2 AWS_PROFILE=profile_account_BBB cosign sign --key awskms:///alias/my_key_in_BBB BBB.dkr.ecr.ap-southeast-2.amazonaws.com/repo/image:tag

Error: signing [BBB.dkr.ecr.ap-southeast-2.amazonaws.com/repo/image:tag]: recursively signing: signing digest: getting fetching default hash function: getting public key: NotFoundException: Alias arn:aws:kms:ap-southeast-2:AAA:alias/DEM_hub_dockersign_key is not found.

Did you see? The key is in BBB account, which is why I specify AWS_PROFILE=profile_account_BBB, yet cosign looks at the key in AAA account.

Even I provide the full arn instead of a key alias, I'll get an authorization issue, because AAA is not authorized to get the key my_key_in_BBB. So again, cosign is using the instance profile instead of using AWS_PROFILE which is set to profile_account_BBB.

Am I missing something? Thanks for your help!

[cpanato]

@joelapatatechaude sorry for the long delay, I was busy with other things

can you try to enable AWS_SDK_LOAD_CONFIG=1 and run that again?

in meantime, I'm trying to get an AWS account (actually two :) ) to try this out

question: you run the sign command in the EC2 machine?

[cpanato]

one question: in your EC2 you use an IAM Role attached to the EC2?