Update Kubernetes integration to support plugin-based architecture for vendor auth

[YihaoOct]

Starting from ~v1.25~ v1.26 of kubectl, vendor-specific authentication code (eg auth against GKE or AKS) will be removed from the core kubectl codebase, in favour of a plugin architecture maintained by the cloud vendors.

The expected release of ~v1.25~ v1.26 will be in October/November 2022.

This means we will need to make sure our steps support these new plugin-based auth mechanisms. We may need to update the worker-tools images.

This may also involve code changes to invoke the new authentication method to avoid errors from using the old authentication (example in the link below).

Link: https://octopusdeploy.slack.com/archives/C020P7U9Z9S/p1654812337731389 (private)

[mjhilton]

The in-tree auth providers were originally slated for removal in kubectl 1.25. Late in the release cycle, a bug was discovered which the kubectl team decided they didn't have time to successfully fix, so the removal of the auth providers was reverted from 1.25 (+ release notes).

The removal of these in-tree vendor-specific auth modules has been moved to the 1.26 branch instead. This gives us a very small bit of breathing room, but we need to get ahead of this deprecation ASAP to ensure we can provide a smooth transition for our customers.

Work towards a GA release of 1.26 is well progressed (GitHub milestone is 85% complete at the time of this comment), so we need to get onto this quickly. It will require:

• Calamari changes to:
  • Ensure compatibility with the new kubectl commands across both Azure and GKE
  • Backported to LTS version of Octopus Server
• Worker Tools changes to:
  • Ensure the vendor plugins are installed alongside kubectl
  • Ensure kubectl version is new enough to have the plugin auth architecture

[mjhilton]

AWS should be unaffected as it already uses the IAM Authenticator

[mjhilton]

The Kubernetes 1.26 release timeline has the following dates slated:

• ✅ 20 September 2022: 1.26.0-alpha.1 release date
• 11 October 2022: 1.26.0-alpha.2 release date
• 1 November 2022: 1.26.0-alpha.3 release date
• 10 November 2022: 1.26.0-beta.0 release date
• 15 November 2022: 1.26.0-rc.0 release date
• 29 November 2022: 1.26.0-rc.1 release date
• 6 December 2022: 1.26.0 GA

We will aim to get this support in place across all LTS branches at least a month in advance of the GA release, which means 6 November 2022 for us.

[mjhilton]

After deep investigation into how our authentication works here, I've discovered that for Azure, we are not affected by this change.

This change would be relevant if the kubeconfig we use had a user block which took this shape:

users:
- name: in-tree-plugin-user
  user:
    auth-provider:
      config:
        apiserver-id: dad814cf-1c1e-4953-b950-c373a821c34f
        client-id: 7a4020c4-c393-4916-9285-0a013ac85eab
        environment: AzurePublicCloud
        tenant-id: 3d13e379-e666-469e-ac38-ec6fd61c1166
      name: azure

In this usage, the "azure" auth provider refers to the old, soon-to-be deprecated authentication code in kubectl. This is being superseded in favour of client-go plugins.

However, our auth works differently. We use the az aks get-credentials command, which:

• Retrieves the details of the cluster
• Generates appropriate user information
• Updates the current kubeconfig with a token/certificate form of auth, of the shape:

users:
- name: clusterUser_mjh-aks-testing_mjh-aks-test
  user:
    client-certificate-data: <cert-data>
    client-key-data: <key-data>
    token: <token>

This means that our Kubernetes cluster auth for AKS does not rely on the authentication approach under deprecation, and does not require any changes. I've tested and validated that a kubeconfig generated by Octopus continues to work with kubectl v1.26-beta.

I suspect the same is true for gcloud, but will investigate and update separately.

[mjhilton]

It appears that gcloud will be affected. For GKE clusters, we use the gcloud container clusters get-credential command, which functionally does the same as Azure's aks get-credentials, however the gcloud version has historically generated a config which uses the in-tree auth approach:

users:
- name: gke_<cluster-name>
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path:<path-to-gcloud>\google-cloud-sdk\bin\gcloud.cmd
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'

The most current version of gcloud (409.0.0) still generates this to-be-deprecated format, unless you provide an env var of USE_GKE_GCLOUD_AUTH_PLUGIN=True, in which case a config compatible with the new approach is generated:

- name: gke_octopus-api-tester_australia-southeast1-c_bold-koala-gke
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: gke-gcloud-auth-plugin.exe
      installHint: Install gke-gcloud-auth-plugin for use with kubectl by following
        https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
      provideClusterInfo: true

[octoreleasebot]

Release Note: Added a warning during deployment for GKE customers if their deployment requires the gke-gcloud-auth-plugin to be available.

[Octobob]

:tada: The fix for this issue has been released in:

Release stream	Release
2023.1	2023.1.2840
2023.2+	all releases