[agent] Kubernetes integration

[xvzf]

• Emit node events (e.g. button press) to k8s
• Control hardware features based on annotations

[xvzf]

Turns out this is quite difficult with talos Linux due to security hardening and lack of /dev/mem support. It'd be wise to move the WS2812 LED logic to a kernel module instead and ship it as extension

[nalum]

Ah, that sucks, I've got talos setup and just hit this issue. Has anyone been pushing this?

Where would be best to get started do you think?

[cedi]

If I had to add my two cents to it, I think there are primarily three parts to the K8s integration:

1. Build a deployment (well, probably daemon-set but with some node anti-affinity rules to prevent the pod from being scheduled on "non compute-blade nodes" in mixed deployments) for the compute-blade-agent on K8s (most likely in the form of a helm-chart) that mounds all the right devices (/dev/mem for example) to the pod so the agent can interact with the hardware correctly.
2. Emit status events of the blade (such as entering/leaving critical mode, turning on/off the identify status, on/off of the stealth mode, ...) to the Kubernetes API as K8s events
3. Integrate the agent config into K8s using CRDs (meaning instead of having to write a ConfigMap with the config.yaml as content you would deploy a CRD object with the Blade Config).

I think 1 and 3 could be integrated quite nicely into one task by using the Operator SDK and the Operator Pattern. Basically I would create a CRD for the config, and if the operator finds the CRD config it would then proceed to deploy a deamon-set in the cluster with that config then mounted to the pod.

Some additional thoughts for the K8s deployment: the agent is now (since #54) sort of stateful. If using the mTLS auth for gRPC to interact with the agents from a remote machine the agent will generate TLS certificates and a bladectl config. You can't mount config-maps RW to a pod. So there needs to be extra precaution with the volumes, so the deployment works in all scenarios