Support EKS Pod Identity

[csantanapr]

Summary

EKS released in Nov/2023 EKS Pod Identity, this is an alternative to IAM roles for service accounts (IRSA)

Motivation

Have additional options on how ArgoCD authenticates with AWS other than using IRSA.

Proposal

To support EKS Pod Identity the change is transparent if your using the aws-sdk, the version of the sdk needs to be updated to a recent version that supports EKS Pod Identity.

[csantanapr]

The PR that bumps the aws-sdk version is https://github.com/argoproj/argo-cd/pull/17063

A follow up PR will have the docs on how to configure ArgoCD to leverage the EKS Pod Identity

[todaywasawesome]

@csantanapr leaving this open until the docs are done

[csantanapr]

Sounds good @csantanapr I will be working on docs

[csantanapr]

@daniel-codefresh I was trying to work on docs but they were broken to build and preview, here is a fix https://github.com/argoproj/argo-cd/pull/17313

[tybook]

Hey @csantanapr do you have a branch with your in-progress docs? Or generally any rough guidance on how to configure ArgoCD to use EKS Pod Identity?

[isikdos]

Hey @csantanapr do you have a branch with your in-progress docs? Or generally any rough guidance on how to configure ArgoCD to use EKS Pod Identity?

I'm not him obvi, but the solution is to just use a more recent release of argo-cd. The only necessary changes to support EKS Pod Identity are to have a more recent version of the AWS SDK running. If you are asking how to configure EKS Pod Identity in general, this guide has a bunch of information: https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

[tybook]

Hey @csantanapr do you have a branch with your in-progress docs? Or generally any rough guidance on how to configure ArgoCD to use EKS Pod Identity?

I'm not him obvi, but the solution is to just use a more recent release of argo-cd. The only necessary changes to support EKS Pod Identity are to have a more recent version of the AWS SDK running. If you are asking how to configure EKS Pod Identity in general, this guide has a bunch of information: https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

I'm wondering specifically about how to configure an EKS cluster secret such that ArgoCD can authenticate with it via EKS Pod Identity instead of IRSA. The existing instructions at https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#eks assume the use of IRSA.

[AnubhavSabharwa]

Does argocd cluster secret support EKS Pod Identity

[magic-madrigal]

I've been trying to find a way to get it work for the past day and a half and haven't been successful.

I feel I'm going to revert to IRSA, and then perhaps try using eksctl's migration command to see what it suggests.

[AnubhavSabharwa]

@magic-madrigal We install argocd via another argocd as helm based application. We are not allowed to save any token on the repository. I have been trying this setup but at the end getting below error

getting credentials: exec: executable argocd-k8s-auth failed with exit code 20

Would you mind suggesting something here? IRSA

[isikdos]

• Have an IAM role that is granted EKS Pod identity with an EKS Pod Identity Association https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_pod_identity_association and don't forget the principal
• Have an IAM role that is given api access to the cluster that you want administrated by EKS Pod Identity. Make sure this can be assumed by the prior role.
• fill out the Argo auth brick for secret-type = cluster : https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Cluster/
• For the above, include the AWS Auth config from here https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/
• Also make sure that you have the tlsclientConfig set properly

If you do all of that, it should work happily, I don't think I've forgotten any steps. You can always make sure of different components by looking at the pod running in EKS and seeing if it has the hallmark signs of using EKS pod identity, and also you can check the logs on the pod identity daemon to see what's going on

Remember to kick pods after updating their configuration for EKS pod identity, as this whole process works when the pod is coming up, not when the pod is already up

[magic-madrigal]

I appreciate the information @isikdos, but the application controller is still requesting additional credentials after going thought your write up.

I'm quite certain that the roles are assuming each other, but they don't seem to be able to get the creds they need to deploy to the cluster.

I've tried going the IRSA route, but yielded similar results.

I just rebuilt my test clusters and I'm going to attempt a slightly different approach.

[AnubhavSabharwa]

@magic-madrigal I tried the IRSA thing, it is looking for additional credentials

[magic-madrigal]

@abhishekjiitr same here. All of the roles are assuming each other; I continue to get an error stating that the cluster is looking for credentials. What Version are you running? I'm on v2.11.7 I'm going to try the newly released v2.12 today.

[magic-madrigal]

@abhishekjiitr - Progress!!!

I updated to 2.12.0, but I don't think this was important to my success.

I followed the IRSA instructions and was still having several issues with my implementation until I discovered an issue with my Cluster Secret. I had mistyped my clusterName in my secret, and once that was correct, it's now connecting correctly.

metadata:
  name: prod-1
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: prod-1
  server: <SERVER_URL_REDACTED>
  config: |
    {
      "awsAuthConfig": {
          "clusterName": "prod-1",
          "roleARN": "arn:aws:iam::<AWS_ACCT_REDATED>:role/Deployer"
      },
      "tlsClientConfig": {
        "caData": "<REDACTED>"
      }
    }

I'm now going to attempt to get this working with Pod Identities, now that I have a working solution

[AnubhavSabharwa]

@magic-madrigal Can you pleae help me out, I tried every way possible but at the end I am getting error

the server has asked for the client to provide credentials OR getting credentials: exec: executable argocd-k8s-auth failed with exit code 20 (Client.Timeout exceeded while awaiting headers)

"tlsClientConfig": { "insecure": "true" }

ARGOCD VERSION: 2.10.10 Please suggest.

When running the command directly on argocd-server pod to add the cluster

argocd-k8s-auth aws --cluster-name test-didev --role-arn arn:aws:iam::123456789:role/argocd-pod-identity-west-2 FATA[0120] timeout while trying to get signed aws request: last error: error presigning AWS request: RequestError: send request failed caused by: Post "https://sts.amazonaws.com/": dial tcp 12.17.16:443: i/o timeout

[mikestef9]

I have Pod Identity working with Argo CD, need to be using at least v2.10

  ArgoCDDeployerRole:
	Type: AWS::IAM::Role
	Properties:
	  RoleName: "ArgoCDDeployerRole"
	  Description: "Role for Argo CD deployment"
	  AssumeRolePolicyDocument:
		Version: '2012-10-17'
		Statement:
		  - Effect: Allow
			Principal:
			  AWS: "arn:aws:iam::123456789012:role/ArgoCDPodIdentityRole"
			Action:
			  - "sts:AssumeRole"
			  - "sts:TagSession"

  ArgoCDPodIdentityRole:
	Type: AWS::IAM::Role
	Properties:
	  RoleName: "ArgoCDPodIdentityRole"
	  Description: "Role to give Argo CD Access"
	  AssumeRolePolicyDocument:
		Version: '2012-10-17'
		Statement:
		  - Sid: "AllowEksAuthToAssumeRoleForPodIdentity"
			Effect: "Allow"
			Principal:
			  Service: "pods.eks.amazonaws.com"
			Action:
			  - "sts:AssumeRole"
			  - "sts:TagSession"
	  Policies:
		- PolicyName: AssumeRole
		  PolicyDocument:
			Version: "2012-10-17"
			Statement:
			  - Sid: "AssumeRole"
				Effect: "Allow"
				Action: "sts:AssumeRole"
				Resource: "arn:aws:iam::123456789012:role/ArgoCDDeployerRole"
				

Then in Helm chart config

configs:
  clusterCredentials:
    ipv4:
      server: "https://redacted.sk1.us-west-2.eks.amazonaws.com"
      labels:
        cluster-name: "ipv4"
      config:
        awsAuthConfig:
          clusterName: "ipv4"
          roleARN: "arn:aws:iam::123456789012:role/ArgoCDDeployerRole"
        tlsClientConfig:
          insecure: false
          caData: "redacted"

[AnubhavSabharwa]

@mikestef9 Thank you for sharing the things, I have same config now but argocd is not able to add the cluster

exec: executable argocd-k8s-auth failed with exit code 20

argocd-k8s-auth aws --cluster-name test-didev --role-arn arn:aws:iam::123456789:role/argocd-pod-identity-west-2 Able to resolve this error by adding region AWS_REGION=us-west-2

Now argocd-k8s-auth aws --cluster-name test-didev --role-arn arn:aws:iam::123456789:role/argocd-pod-identity-west-2 gives me the token however, if i only run argocd-k8s-auth aws --cluster-name test-didev also returns the same thing. Running in argocd-server pod Not sure whether the arn is required or not. However, when deploying the secret it is giving timeout while getting the credentials which is totally not understandable.

the server has asked for the client to provide credentials

This is the error.

aws-auth for target cluster has been updated. Can someone please help me what am I missing?