CI/CD: Switch to lightweight Kubernetes distribution

[arm4b]

Currently, we're relying on CircleCI large container size that provides 4CPUs & 15GB RAM which is a paid feature we won't be able to afford in the future: https://github.com/StackStorm/stackstorm-ha/blob/3ba85e6c584229e21cf41e2d19ec14f2bef4ffcc/.circleci/config.yml#L52-L55

1. Switch to a smaller CircleCI container size
2. Replace K8s with a lightweight distribution like K3s or K0s that'll fit CI/CD environment better:

• https://k3s.io/
• https://github.com/k0sproject/k0s/
• https://kind.sigs.k8s.io/

Hopefully, this could also help with the issues when builds reported random stability failures caused by etcd/K8s/CircleCI.

[cognifloyd]

This looks like a good way to run k3s in Github Actions: https://github.com/marketplace/actions/actions-k3s

[cognifloyd]

Or this for kind: https://github.com/marketplace/actions/kind-cluster

[arm4b]

Adding an update here as some things are outdated now (sorry for that!). We're using CircleCI for free these days and the container we're using is large as well: https://github.com/StackStorm/stackstorm-k8s/blob/b45c85937e08f97cd697e66d764763835fb9e1a3/.circleci/config.yml#L50-L55

so no issues with CircleCI running minikube + k8s + helm for now as that probably provides the most consistent K8s testing experience.

But perhaps having an additional run in K3s to see if it works makes sense too?