[RFC] declarative Kubernetes API usage pattern

[errordeveloper]

trafficstars

The nature of some of the functionality that cilium program implements, is often very procedural, however that shouldn't imply that all of the logic around API interactions is also procedural.

Presently, cilium install and cilium connectivity check follow a procedural pattern where a set of objects is generated and written to the API directly.

https://github.com/cilium/cilium-cli/blob/dff492aeee540d72c27ebc819d3ad522382e6b60/install/install.go#L1386-L1512

https://github.com/cilium/cilium-cli/blob/dff492aeee540d72c27ebc819d3ad522382e6b60/connectivity/check.go#L827-L969

This is largely dictated by the Go client which requires calling a very specific function for each object type. However, it is not necessary to use this client for all API interactions.

• it would desirable to be able to serialise all objects to YAML or JSON for user to apply separately, e.g. using GitOps, or just so they can preview what would happen (--dry-run)
• it would be quite convenient if the implementation operated explicitly on a set of objects, rather then it being function with many side-effects, which also make the code more easily unit-testable
• it would be desirable to have a single function that handles create-or-update logic that can leverage server-side-apply in clusters where it is available, as opposed to having a function for each object type
• it should be possible to interact with some APIs dynamically, i.e. without having to include a fully-typed client that has extra dependencies and being bound to a particular version of the give API, e.g. this could be useful to support multiple version of Cilium APIs, or enable integrations with e.g. OpenShift APIs without needing a client with full set of types

This can implemented fairly easily using http://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/client, which can handle typed and unstructured objects.

Here is one example (albeit it m only does create-or-skip, it can be easily extend to support create-or-update/apply):

https://github.com/isovalent/gke-test-cluster-operator/blob/a6824ed4a47b5c0f78b1aefccfda39c7ef363b7e/controllers/common/common.go#L73-L114

[errordeveloper]

There is also a library that implements server-side-apply functionality with a fallback to client-side implementation that uses same code as kubectl apply does.

[xiaods]

cool, need pr welcome