digitalocean kubernetes: kubeconfig and access tokens

[ulrichSchreiner]

hi,

when creating a kubernetes cluster on digitalocean the pulumi state contains the kubeconfig for the created cluster. this is a problem with digitalocean because they:

• create a new api token on every download of the kubeconfig
• expire the token after 7 days

so i have the problem that a pulumi up works only within the first seven days. after that a pulumi up shows the cluster as not accessible and marks the resources in the cluster as deleted.

when you do a pulumi refresh a new DO API token is generated. every refresh generates a new token. after a handful of refreshes you have a handful of api-tokens with DO (and they will all disappear after seven days).

i'm not sure what the solution should look like. but it looks not perfect to store the KUBECONFIG in the state when this kubeconfig will fail after seven days. perhaps you should explictly retrieve a config with a higher expire-value? or give the user an easy way to refresh/up the stack when the kubeconfig does not work any more.

more info: https://www.digitalocean.com/community/questions/when-saving-kubernetes-config-from-doctl-with-an-existing-api-token-another-token-is-created-is-this-normal

[leezen]

Related: https://github.com/pulumi/pulumi-digitalocean/issues/78

[metral]

Cross posting comments below from https://github.com/pulumi/pulumi-digitalocean/issues/78

---

I've been using DOKS 1.16 and 1.17 this week, and have not experienced the issue with certificate-authority-data changing on pulumi refresh or pulumi update.

Per DO engineers, newer versions of k8s (starting with 1.16) by default use DO Access Tokens instead of certificates for the admin user's kubeconfig.

DO automatically creates this token for the cluster admin's kubeconfig, and the token expires after 7 days. However, the certificate-authority-data has a default expiration of 20 years -- (this could be a new change for the ca data not changing?)-- for example, here's the openssl x509 text output on a cluster's ca data created today:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1653 (0x675)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = DigitalOcean, CN = k8saas Cluster CA
        Validity
            Not Before: Jun  5 16:39:34 2020 GMT
            Not After : Jun  5 16:39:34 2040 GMT
        Subject: O = DigitalOcean, CN = k8saas Cluster CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
....

This comment indicates that you can form your own kubeconfig using a normal DO API token, and disregard the automatic one created by DO. Using your own API token over the automatic one created seems favorable as the automatic one is set to expire in 7 days.

The pulumi equivalent of using a user-provided DO API token in a kubeconfig can be done using the following, e.g. in TS:

// Manufacture a DO kubeconfig that uses a given API token.
//
// Note: this is slightly "different" than the default DOKS kubeconfig created
// for the cluster admin, which uses a new token automatically created by DO.
export function createTokenKubeconfig(
    cluster: digitalocean.KubernetesCluster,
    user: pulumi.Input<string>,
    apiToken: pulumi.Input<string>,
): pulumi.Output<any> {
    return pulumi.interpolate`apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${cluster.kubeConfigs[0].clusterCaCertificate}
    server: ${cluster.endpoint}
  name: ${cluster.name}
contexts:
- context:
    cluster: ${cluster.name}
    user: ${cluster.name}-${user}
  name: ${cluster.name}
current-context: ${cluster.name}
kind: Config
users:
- name: ${cluster.name}-${user}
  user:
    token: ${apiToken}
`;
}

[iyobo]

@metral What is the "user" parameter in this function?