[FEATURE] Add possibility to upgrade a node

[fastlorenzo]

Is your feature request related to a problem or a Pull Request

No

Scope of your request

It would be great to be able to upgrade nodes without re-creating the entire cluster. One possibility would be to add an --image XYZ flag within the node edit functionality (similar to the --port-add one).

Describe the solution you'd like

It might be possible to use the logic from node create --image XYZ, where the node edit command would copy the specs of the node to edit, shut it down and remove it, then patch the image version and re-create it.

Describe alternatives you've considered

I've tried using docker-compose to setup the cluster, but this doesn't work. Also looked at watchtower, but no luck there.

[iwilltry42]

Hi @fastlorenzo , thanks for opening this feature request! In general, this would be the same as k3d node delete, followed by k3d node create, right? Still, the state of the node will be lost and there's no way for "zero downtime" upgrades here.

So it's super difficult and we already tried many different ways to upgrade nodes while keeping the state, but (especially for server nodes), it's close to impossible to have a clean solution.

I'm definitely open for ideas if you have any :)

Anyway, I'd like to know, what you'd expect to see here apart from replacing the container with a new container using a newer image? (same labels, same port-mappings, etc. would be possible)

[fastlorenzo]

understood, in my case, replacing the container with the latest image and the same options would be ideal. This should be similar as upgrading the ´k3s´ executable version when performing an upgrade. I could do it within the running container, but that goes against immutability principle of containers 😄

[oursland]

Hi @fastlorenzo , thanks for opening this feature request! In general, this would be the same as k3d node delete, followed by k3d node create, right? Still, the state of the node will be lost and there's no way for "zero downtime" upgrades here.

I believe the process is to:

1. launch an upgraded node with the same configuration as the node you wish to replace (labels, taints, etc).
2. run kubectl drain --ignore-daemonsets <node name> to force the jobs to be rescheduled, ideally to the newly added node, but not necessarily.
3. once the node has completed the drain request, terminate it.

[atroxix]

I'm sorry to ask this trivial question, but is there an "official" procedure to upgrade k3d clusters/nodes? I cannot find information anywhere about it. I'd imagine it would be a relevant one for FAQ.

Do I understand it correctly that all data (without any volume mappings) is created within the nodes aka docker containers, so when a node is removed (to be replaced) then data is removed too? If so then what are the files and directories necessary to be mapped/preserved when replacing a node with an updated one?