Add support for Cloud Controller Manager (addon?)

[headcr4sh]

Feature Request

Add support for the Cloud Controller Manager (possibly as Typhoon addon?) Documentation: https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/

Feature

Kubernetes v1.6 introduced a new binary called cloud-controller-manager. cloud-controller-manager is a daemon that embeds cloud-specific control loops. These cloud-specific control loops were originally in the kube-controller-manager. Since cloud providers develop and release at a different pace compared to the Kubernetes project, abstracting the provider-specific code to the cloud-controller-manager binary allows cloud vendors to evolve independently from the core Kubernetes code.

The cloud-controller-manager can be linked to any cloud provider that satisfies cloudprovider.Interface. For backwards compatibility, the cloud-controller-manager provided in the core Kubernetes project uses the same cloud libraries as kube-controller-manager. Cloud providers already supported in Kubernetes core are expected to use the in-tree cloud-controller-manager to transition out of Kubernetes core. In future Kubernetes releases, all cloud controller managers will be developed outside of the core Kubernetes project managed by sig leads or cloud vendors.

(source: https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/)

Tradeoffs

Pros

• Cloud provider specific functionality (in my case: access to Elastic Container Registries inside AWS) can be achieved without the deprecated --cloud-provider flag that can be set on kube-apiserver and kube-controller-manager

Cons

• Additional complexity is introduced, because kubelets should be invoked with the flag --cloud-provider=external, which indicates, that no workload should be scheduled on nodes prior to full initialization of the cloud controller manager

[dghubble]

Doesn't the --cloud-provider=external flag preclude this being an addon? It would have to be maintained in core. I haven't spent much time exploring whether CCM is ready for prime time yet.

[devurandom]

Digital Ocean support would probably benefit from switching to the external cloud provider and using cloud controller managers, since it has no built-in cloud provider, while a Digital Ocean cloud controller manager exists.

See-also: https://github.com/poseidon/typhoon/pull/422#issuecomment-472121185

[dghubble]

This is still on ice as long as CCM is at odds with Kubelet TLS bootstrap (which Typhoon has been inching toward, though no final decision).

For DigitalOcean, their CSI provider is a nice way to enable cloud-specific storage without any cluster modifications (the poster child of doing this right). But that leaves CCM offering load balancers (non-goal, there's always a better option than service type=LoadBalance) and labels/addresses. Only vaguely useful.

Ideally, CCM (generally) would not inhibit future choices around TLS bootstrap, DO account tokens would get some reaonsable scoping (having one in a secret would drastically weaken the current security posture), DO CCM would be reasonably mature, and have the ability to disable load balancer. Looks promsing imo, but not there yet.

[dghubble]

I think at this point, any cloud provider specific controllers are managed and/or documented by their respective cloud providers. It's probably not in-scope for this project to directly integrate these.