[FEATURE] Distributed Clusters

[zewelor]

Scope of your request

Ability to create node, and add it to already existing cluster, on other computer.

Describe the solution you'd like

When I create cluster via k3d/k3s, I would like to be able to add workers to it, from other computers, using k3d ( So that workers will be inside docker containers ).

[iwilltry42]

Hi @zewelor , thanks for opening this feature request. Can you describe your use-case, what you want to achieve? k3d is mostly meant for local development on and with Kubernetes, so possibly this is rather a use-case for plain k3s?

[zewelor]

Hi

Thanks for reply !

I was thinking about building some home cluster, with devices like qnap ( there is docker support but adding custom software is somehow limited ), some rpi with kodi only distibution ( there is docker addon, but adding some other custom software is rather hard ). Maybe my desktop ( which I would prefer to keep "clean" so some worker inside docker would be nice ).

I had impression ( from reading issues for versions < 3.x) that its supported, now I see its not longer valid for 3.x.

I could setup master on plain k3s, but workers on docker would be really good. Maybe as you wrote in other comment, doing master in docker and connecting to it from other dockerized workers is too complicated. So maybe plain k3s as master, with dockerized k3s workers ? Or maybe I should look for other solution ? Or maybe i should skip k3d and use https://hub.docker.com/r/rancher/k3s/tags ?

[iwilltry42]

I get your use-case. I wouldn't really recommend using k3d for such a "production-like" use-case due to its limitations introduced by the additional docker layer. You will always hit limitations like wanting to expose additional ports after creating the cluster/nodes, etc. Since you're planning to run one k3d node per physical host, it could actually work network-wise, because you can simply create the nodes with --network host and have them act like any other process on your host machine (network-wise) :+1:

You're right, that this was possible with k3d v1.x, because there we didn't have auto-detection of existing k3d clusters and supporting such use-cases was not really in scope back then. But I also like the use case and I think this could land in v3.1.0 :+1:

Apart from that, there's an example docker-compose file in the k3s repo, that you could adjust to your needs (https://github.com/rancher/k3s/blob/master/docker-compose.yml).

[zewelor]

Thanks for thorough explanation. I will try with docker-compose example from k3s repo.

[andrewjameshough]

Hi there - i'd like to add a little context to this request as i have a similar requirement. Current setup is a cluster of raspberrypis (4 node k3s) as my main cluster which works fantastically well for 95% of the time - the trouble is i need an x86 node for certain images / tasks. Instead of having to purchase another SBC that maybe inferior to what i have i'd like to be able to add a k3d node through my iMac / a.n.other machine and use the architecture label to distribute specific x86 tasks to that node.

Been following the enhancement progress on this and would be very grateful for this to be added!

Many thanks.

[blackliner]

Hi, I would also be interested in a clean way to add and remove worker nodes in an existing cluster. My setup looks like this:

• Proxmox cluster with n nodes on them (currently 3) and they run on some old commodity HW (quadcore intel i7)
• Main Workstation with AMD R3900X@64Gb RAM that could supply some CPU and RAM to the cluster, running W11 and WSL2

So instead of having to natively install k3s, I would really like to see a way to "just attach" a node to an existing cluster temporarily. Or just add a second master, take the other offline, replace it with a new one and take the temporary down again (no idea if that works though :D ) Thanks!

[guhuajun]

Greetings, Just come from this PR, https://github.com/rancher/k3d/pull/734.

I am trying to challenge myself for setting up a simulated cross-region deployment with limited resouces, a Dell T3431 workstation with 4 cores and 64GB RAM. K3D is very handy when you only have 4 cores. I am using VirtualBox with 3 VMs.

Network:

* 192.168.0.0/24

Host:

* 192.168.0.13

Local Registry and HAProxy Server (a simualted GSLB) :

* 192.168.0.31

VMs with bridge network:

* 192.168.0.41 (Region: Shanghai)
* 192.168.0.42 (Region: Beijing)

There are 3 clusters, 2 k3d clusters for applications in each region, 1 K3S cluster for middleware and databases across regions. So that's the motivation for me to add one node to an exsting clsuster. But since K3D is not ready for this scenario, I am using K3S instead. Meanwhile I am using supervisord for managing this K3S cluster.

I am using following code to manage k3s cluster. (Python click + fabric) If someone is interested in this private repo (code is still dirty), please let me know. Actually this is an upgrad project for my public repo, https://github.com/guhuajun/kong-keycloak-playground

@k3s.command()
@click.option('--action', type=click.Choice(cluster_actions, case_sensitive=False))
@click.option('--dry-run', type=bool, default=False)
def cluster(action, dry_run):
    click.secho('Please run server init if some files are missing.', fg='green')
    click.secho(f'K3S: {action} cluster ...', fg='green')

    if action == 'create':
        _put(['192.168.0.41'], 'k3s/k3s-server.ini',
             '/etc/supervisord.d/k3s.ini')

        commands = [
            'supervisorctl reload',
            'supervisorctl status all'
        ]
        _run(['192.168.0.41'], commands, workspace='/root/lab/', dry_run=dry_run)

        click.secho('Sleeping 40 seconds for starting api server ...')
        time.sleep(40)

        commands = [
            'cat /var/lib/rancher/k3s/server/node-token',
        ]
        agent_token = _run(['192.168.0.41'], commands,
                           workspace='/root/lab/', dry_run=dry_run, output='script')
        agent_token = agent_token.strip('\n')

        _put(['192.168.0.42'], 'k3s/k3s-agent.ini',
             '/etc/supervisord.d/k3s.ini')

        commands = [
            f'''sed -i 's/TOKEN/{agent_token}/g' /etc/supervisord.d/k3s.ini''',
            'supervisorctl reload',
            'supervisorctl status all'
        ]

        _run(['192.168.0.42'], commands, workspace='/root/lab/', dry_run=dry_run)

    if action == 'restart':
        commands = [
            'supervisorctl restart k3s'
        ]

        _run(servers, commands, workspace='/root/lab/', dry_run=dry_run)

    if action == 'delete':
        commands = [
            'rm -rf /etc/supervisord.d/k3s.ini',
            'rm -rf /var/lib/rancher/k3s',
            'supervisorctl reload',
            'supervisorctl status all',
            # 'systemctl restart supervisord'
        ]
        _run(servers, commands, workspace='/root/lab/', dry_run=dry_run)

Finally, K3D is awesome! ;)

[b2az]

Why I would have a need for distributed k3d installations: Our sysadmins are giving us machines with docker preinstalled and a technical user which is allowed to execute docker commands and some basic os ones. No installation of third party software is allowed.

[fzyzcjy]

Looking forward to this feature!