[Feature] Bootstrapping the whole cluster with single command

[jnummelin]

Expected Behaviour

Currently k3sup requires user to bootstrap the server first and then join nodes separately. Would something like this make sense:

$ terraform output -json | yq r - k3s_cluster.value > cluster.yml && k3sup cluster --config cluster.yml 
$ #wait for ~2mins
$ kubectl get node
--> cluster online

In this example the cluster.yml file could look something like:

server:
  ip: 95.217.129.189
  user: root
workers:
- ip: 95.217.129.185
  user: root
- ip: 95.217.129.139
  user: root
- ip: 95.216.210.52
  user: root

(of course it should support all the options k3sup install and k3sup join supports)

As in the example command, this sort of approach would make working with terraform based infrastructure a breeze.

Note: I've got a super rough prototype of this mostly working, just wanted to see if this in general would be something that might get accepted into k3sup directly.

Current Behaviour

User needs to run a command per node, either k3sup install or k3sup join depending on the node role.

Possible Solution

Have a cluster config file that can easily be used to describe the whole cluster and the a command like k3sup cluster --config cluster.yml to bootstrap the whole cluster at once.

Steps to Reproduce (for bugs)

Context

Your Environment

• What OS or type or VM are you using? Where is it hosted?

• Operating System and version (e.g. Linux, Windows, MacOS):

[alexellis]

Hi @jnummelin thanks for your interest in k3sup, nice to hear from you.

I've thought about YAML files before, but didn't want to suggest it as I think it can quickly get out of hand having offered YAML and CLI flags in OpenFaaS. I might still be convinced if I get a bit more time to work on this project. I wonder what @ibuildthecloud thinks?

I was wondering about an all-in-one approach though, with or without YAML, perhaps using Ansible since that's a good match for ssh. https://galaxy.ansible.com/vandot/k3sup

How would your terraform script work, do you have an example you can share?

Alex

[jnummelin]

Well, having a declarative way to describe your cluster and bootstrap/manage it has been proven to work with many k8s distros. And as a pattern it's very familiar for people coming to k3s from other k8s distros. Also in this case it makes perfect sense, at least in my head :), as k3sup is also able to setup many add-on components. So with a yaml one could define not only the cluster nodes, but also the add-on components and their configuration.

One use case where this kind of simple approach to bootstrap the cluster with oneliner is in automated testing pipelines. So as part of your testing, you could now super easily bootstrap a small "throw-away" test cluster and deploy your app on it automatically.

As I mentioned, I've got a working prototype of this already which uses this kind of terraform:

variable "hcloud_token" {
    description = "Hetzner API token"
}

provider "hcloud" {
  token = "${var.hcloud_token}"
}

variable "ssh_keys" {
    default = []
}

variable "ssh_user" {
    default = "root"
}

variable "cluster_name" {
    default = "k3sup"
}

variable "location" {
    default = "hel1"
}

variable "image" {
    default = "ubuntu-18.04"
}

variable "master_type" {
    default = "cx11"
}

variable "worker_count" {
    default = 1
}

variable "worker_type" {
    default = "cx11"
}

resource "hcloud_server" "master" {
    name = "${var.cluster_name}-master"
    image = "${var.image}"
    server_type = "${var.master_type}"
    ssh_keys = "${var.ssh_keys}"
    location = "${var.location}"
}

resource "hcloud_server" "worker" {
    count = "${var.worker_count}"
    name = "${var.cluster_name}-worker-${count.index}"
    image = "${var.image}"
    server_type = "${var.worker_type}"
    ssh_keys = "${var.ssh_keys}"
    location = "${var.location}"
}

output "k3s_cluster" {
    value = {
        server = {
            ip    = hcloud_server.master.ipv4_address
            user  = "root"
        }

        workers = [
            for host in concat(hcloud_server.worker)  : {
                ip           = host.ipv4_address
                user         = "root"
            }
        ]
    }

}

(bit of mix-and-match of old and new syntax, ignore that for now :) )

So essentially with the direct terraform output, translated to yaml with yq, one is able to easily setup the whole cluster at once. And essentially you have only one place where you define the cluster and that's terraform. But rather than building more deep/direct integration with Terraform, the cluster.yml is used as the "interface" which makes also the cluster.yml easily usable without Terraform.

I've taken the PoC already far enough to be able to bootstrap the workers in parallel, thus minimizing the cluster (1+3) bootstrap time to less than two minutes on low-end VMs in Hetzner. :)

[Kerwood]

This is definitely a wanted feature. This would making new clusters much easier by just reusing your personalized cluster definition file. Also it would make sense being able to provision your cluster the same way as RKE, except this would be much cooler being able to use the output from Terraform.

[Kerwood]

Sooo... Is this going to be a thing or ?

[JElgar]

Hi! This sounds like a great feature! I was thinking of working on a similar thing for a personal project. If @jnummelin isn't able to follow this up, I'd be happy to have a go/help out?

[nicklasfrahm]

@alexellis Can I submit a PR for this or is this not wanted at all?

[Kerwood]

I think @jnummelin already has a prototype on this?

[jnummelin]

Well, I had a working proto for this but that's long gone in trash. :D So others are free to take over if this is of interest for folks. Originally I did not get clear signal whether this would be of interest at all.

[nicklasfrahm]

For anyone still interested in this, I went ahead and started a seperate project for this over at nicklasfrahm/k3se. Pretty alpha at the moment as the version indicates and no HA support but I am on it.

[alexellis]

We may consider this for a k3sup pro in the future, but it's not on the roadmap for the free tool, as it is nobody here who's asking for features is sponsoring me.

Alex

[alexellis]

/lock: wontfix