Add support for kubeconfig (CLI) integration

[jcogilvie]

Is your feature request related to a problem? Please describe. I am a user of kubernetes and of kubectl and eks. At present, kubectl references the aws binary for authentication, which expects certain named profiles to exist based on my kubeconfig file.

Without Leapp installed, kubectl finds the profile and delegates to the aws binary which correctly prompts me for MFA and then works.

With Leapp installed, I need to pre-authenticate the session in Leapp or else the profile is not found. This represents a usability regression from the default awscli case.

Describe the solution you'd like Instead of kubectl directly asking AWS, there should be a way for kubectl to ask Leapp.

It appears that a kubeconfig file (specifically one generated by aws eks ...) specifies how authentication is done by referencing the aws binary:

- name: arn:aws:eks:us-east-1:...:cluster/some-cluster-name
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - some-cluster-name
      command: aws
      env:
      - name: AWS_PROFILE
        value: an-aws-config-profile
      interactiveMode: IfAvailable
      provideClusterInfo: false

It seems like integration could be achieved with a rewrite of the kubeconfig file to call Leapp on the command line rather than aws.

So, Leapp needs to:

1. provide such command line functionality as necessary for this integration (including a way to prompt me for MFA tokens when the integration is called, either in the terminal session or in a GUI prompt)
2. provide documentation for how to rewrite kubeconfig to support the use case

Describe alternatives you've considered An alternative would be to keep the named profiles in the aws config to retain the existing awscli functionality, but this presents challenges with spawning an MFA prompt when necessary.

I also imagined a case where Leapp could parse a kubeconfig file like the one above and rewrite it, but that seems out of scope for an app like this as it's kubernetes-specific.

[pethron]

Thanks for reporting! I'm pleased to see some interest in k8s and Eks, and this could be a good starting point for Leapp. Your suggestion opens a few topics and solutions:

From what I see from the kubeconfig file the command used is the "eks get-token" and set the AWS_PROFILE to let the awscli generate the correct set of credentials through the credential provider. This means that for directly supporting kubectl we need to integrate with the eks API (which we'd like to do, but right now goes a bit out of scope from Leapp).

A more feasible alternative would be to let that resolution fall entirely on the awscli trough sourcing credentials with an external process. We have an open discussion on this and I think it's totally doable.

For this and other reasons, we are moving Leapp's logic to a daemon that will support even a CLI. You can find the repository here. The external source process leaves the kubeconfig file completely agnostic, under the hood will be the awscli to call directly to leap.

In the future, we can directly integrate with the kubeconfig file, but for now, I think this solution is reasonable. What do you think? If you'd like to contribute to the daemon and CLI feel free to come to our slack channel.

[ericvilla]

Hi @jcogilvie,

looking at the docs, aws eks get-token or aws-iam-authenticator rely on credentials provided by default AWS credential provider chain, unless you specify AWS_PROFILE env variable in kubeconfig file.

As far as now, I think your configuration should work by setting AWS_PROFILE env variable to the profile you defined inside Leapp. Leapp provides a default AWS Named Profile that can be associated with the sessions. You can add another AWS Named Profile and reference it from the kubeconfig file.

With Leapp installed, I need to pre-authenticate the session in Leapp or else the profile is not found. This represents a usability regression from the default awscli case.

If I understood what you mean, you'd like a way to generate credentials "on-demand" when you issue kubectl commands. As @pethron stated, sourcing credentials from an external process could be the best fit for this need. We have an issue open about this topic, in addition to the previously linked discussion.

[jcogilvie]

If I understood what you mean, you'd like a way to generate credentials "on-demand" when you issue kubectl commands. As @pethron stated, sourcing credentials from an external process could be the best fit for this need. We have an issue open about this topic, in addition to the previously linked discussion.

This sounds promising as a direction for development.

The CLI hook you're looking for might be as simple as inserting yourselves in the config file and proxying the call to aws eks underneath:

- name: arn:aws:eks:us-east-1:...:cluster/some-cluster-name
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --whatever-args-are-necessary-to-get-an-eks-token
      command: leapp   # a leapp CLI executable
      env:
      - name: AWS_PROFILE
        value: an-aws-config-profile
      interactiveMode: IfAvailable
      provideClusterInfo: false

[pethron]

Yeah, but we're trying to limit the interaction with other services for a design choice. My suggestion was to leave the kubeconfig exactly like that and instead make the awscli compatible with Leapp. This would make Leapp compatible with any tool that can outsource the credential process (instead of only eks and kubeconfig)

[jcogilvie]

That definitely sounds like a bigger win, yes. Also perhaps a bigger lift.

I'll be happy as soon as my use case is solved, and you're the expert. 😁

[ericvilla]

I'm not sure, but I think that - as far as know - it should still work by manually setting the AWS_PROFILE (in the kubeconfig file) to the one associated with the active Leapp session.

Probably, we could let kubectl rely on the default AWS credential provider chain, and sourcing credentials using the Leapp CLI as described here, i.e. from the .aws/config file.

[jcogilvie]

I'm not sure, but I think that - as far as know - it should still work by manually setting the AWS_PROFILE (in the kubeconfig file) to the one associated with the active Leapp session.

Today, yes, this is what I do. I have 8 kubeconfig entries each with their own associated aws profile. Each profile is built on an assumed role from one central account with MFA on it. So for now, I open leapp, click the associated role for the kubectl context I need, enter my mfa if it isn't already cached, and then I can use the CLI.

I try to never keep production credentials in my default profile, in case of an accidental command input from myself or some other application that uses the default profile.