kops
kops copied to clipboard
kubernetes
Windows node support
From 1.5 we have windows containers support in kubernetes Can kops be extended to support windows containers
Admittedly that is a bunch of work. We always love contributions! If we can find someone to take it on the work, I am all for it!!!
First thing we need a cross build that runs in windows.
Docker supports running windows containers on windows host, windows containers can't run on linux host. https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/ - more about windows containers.
currently all the nodes that are created are linux based, it means windows container cant be run on it. a configuration/ flag to say how many windows nodes you want would be great to deploy both types of containers.
@ahasnaini at this point windows nodes are not supported. It would be work to have nodeup compatible with windows, but it is not impossible. If anyone is interested in implementing the support for windows nodes we are happy to provide guidance!
@ahasnaini I updated the title of the issue to represent your request for windows node support. Please re-word as needed.
Can someone provide steps to manually setup kubernetes using AWS EC2 instances on windows container. I tried to setup linux host and windows host but no luck. I would be great if some one provide steps to setup kubernetes master node on ubuntu and worker nodes on windows container.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Prevent issues from auto-closing with an /lifecycle frozen comment.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale
@chrislovecnm could you give a few words on what needs to happen at a high level in order for this to work? I'm assuming it'd be supported on k8s 1.9+. Just want to get an idea which part of kops would be subject to change.
@perryao nodeup is our component that bootstraps a node, and it would need to be updated greatly. Give you an idea all of the models are task components that nodeup completes. Here is the directory with the models https://github.com/kubernetes/kops/tree/master/nodeup/pkg/model. It is also completely driven by systemd, so we would need to tweak that for windows.
If you want the nitty-gritty details here is the main command loop for nodeup https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/nodeup/command.go
Let me know if you want a more detailed walk-through.
/lifecycle frozen /remove-lifecycle stale
Is there windows node preview brach can share? I would like to contribute my kubeadm + windows experience. Thanks
At the moment there is no plan to add Windows support to Kops. Not to say that we don't want to have it, but none of the active maintainers expressed any interests. As it was written above, this is no trivial task by any means and anyone that want to work on this would have to have experience with the code base.
The most advanced work that I know of into making Kops work with windows is here, in case anyone is interested: https://github.com/geodata-no/windows-kops-nodeup
To add to @hakman's comment, with the recent ARM64 support the steps towards adding windows support have been simplified a bit but it will still require significant changes. In case anyone is interested in contributing windows support, these are the high level concerns we'll need to address:
- UserData - Since the userdata will need to be entirely different, we'll need to rely on a new API field in the InstanceGroupSpec that instructs the kops cli to generate windows userdata. We'll need to "convert" all of the linux userdata in bash to powershell, which it looks like windows-kops-nodeup does.
- nodeup - We'll need to build windows versions of nodeup. This should be pretty straight forward. The trickier part will be modifying all the nodeup code to be os-agnostic.
- Any filepaths should use the "path/filepath" package to make them windows compatible
- File permissions and ownership are significantly different on windows (no UID/GID) so any file-related nodeup tasks will need to be updated
- nodeup installs a variety of dependency packages (docker, ntp, etc.), we'll need to figure out if/how we do that on windows.
- nodeup sets up services that run in the background (using systemd for example) that will need porting to windows.
- CNI support - we'll need to determine which CNIs support windows nodes, adjusting the daemonset manifests as necessary.
- testing - we'll want to have e2e test coverage with windows which will require some changes to our test infrastructure. Some of it (kubetest - which shells out the kops commands as well as SSH's into nodes to download their logs) is in "maintenance mode" right now so adding windows support might be difficult.
This is probably a good reference: https://github.com/microsoft/SDN/blob/master/Kubernetes/containerd/start.ps1
I'm sure there are other things I've missed, and I don't have any windows k8s or windows cloud experience at all so anyone else with other concerns can feel free to add them here.
The contributors will be more than happy to review any PRs that work towards Windows support though, and attending the Kops office hours would be a great way to hash out any of these concerns :)
@andersosthus , a co-worker of mine, is the author of https://github.com/geodata-no/windows-kops-nodeup. We do have windows worker nodes working with kops 1.17. Internally we consider Windows support as beta in our clusters.
The company I work for can contribute with adding windows support to kops if there is general interest in this. @andersosthus will be our main resource, and can comment him self on the current state of things.
proof of working windows node ;)
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-4-134-234.eu-north-1.compute.internal Ready master 37d v1.17.5 10.4.134.234 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-136-246.eu-north-1.compute.internal Ready node,spot 34d v1.17.5 10.4.136.246 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-146-239.eu-north-1.compute.internal Ready node,spot 15d v1.17.5 10.4.146.239 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-147-178.eu-north-1.compute.internal Ready database,node 69d v1.17.5 10.4.147.178 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-152-236.eu-north-1.compute.internal Ready node,storage 69d v1.17.5 10.4.152.236 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-153-120.eu-north-1.compute.internal Ready node,spot 2d23h v1.17.5 10.4.153.120 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-158-65.eu-north-1.compute.internal Ready node,spot 15d v1.17.5 10.4.158.65 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-165-117.eu-north-1.compute.internal Ready node,spot 34d v1.17.5 10.4.165.117 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-165-150.eu-north-1.compute.internal Ready node,spot 39d v1.17.5 10.4.165.150 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-171-175.eu-north-1.compute.internal Ready master 69d v1.17.5 10.4.171.175 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-171-221.eu-north-1.compute.internal Ready database,node 69d v1.17.5 10.4.171.221 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-182-173.eu-north-1.compute.internal Ready node,spot 29d v1.17.5 10.4.182.173 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-184-40.eu-north-1.compute.internal Ready node,windowsspot 69d v1.17.5 10.4.184.40 <none> Windows Server Datacenter 10.0.18363.720 docker://19.3.5
ip-10-4-190-145.eu-north-1.compute.internal Ready node,storage 40d v1.17.5 10.4.190.145 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-198-87.eu-north-1.compute.internal Ready node,spot 30d v1.17.5 10.4.198.87 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-205-193.eu-north-1.compute.internal Ready database,node 69d v1.17.5 10.4.205.193 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-209-75.eu-north-1.compute.internal Ready node,spot 29d v1.17.5 10.4.209.75 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-213-0.eu-north-1.compute.internal Ready node,spot 28d v1.17.5 10.4.213.0 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-217-72.eu-north-1.compute.internal Ready node,storage 40d v1.17.5 10.4.217.72 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
ip-10-4-222-161.eu-north-1.compute.internal Ready master 69d v1.17.5 10.4.222.161 <none> Ubuntu 20.04 LTS 5.4.0-1009-aws docker://19.3.4
Hi,
As @paalkr mentioned we do have a working set of scripts for Windows Support. Currently we have a bug with DNS resolution, but I expect that is due to a misconfiguration of CNI. Also, as mentioned in that repo, our code is based on the work CCPGames have done in their Kops Windows repo over at https://github.com/ccpgames/platform-kube-windows-nodeup, but adjusted for Kops 1.17.
Based on earlier discussions, I got the impression that "official" Windows support in Kops wouldn't come until the Kubernetes Cluster API was stabilized and implemented, since that should handle Windows as well, but if things have changed, I am happy to contribute to this. Updating nodeup to use path/filepath, and ensuring that all the default DS/Deployments that Kops configures is configured with OS tolerations.
Also, in regards to CNI, as far as I know, only flannel is supported on Windows, and to get flannel to work, we have to use a different VNI and Port than the default due to Windows limitations.
I've also started to look into running some of the components as pods as opposed to running them directly on the node. I think I saw someone else do that approach, which would make this a bit easier.
I'll try to drop in on the next Office Hours and see if we can maybe have a chat about it.
Any update on this? Are you guys working on it and you're planning to release at some point?
I would love to see this feature released. I can help with testing at least.
Cheers.
The basic functionality works, and we run windows workloads in our clusters. But there is a great potential for integrating windows support more tightly into Kops.
The biggest obstacle for us right now is all the general limitations with Kubernetes on Windows. Especially storage (CSI) is hard, but none of this issues are related to Kops in particular, they apply to Kubernetes on Windows in general.
We've stopped working on this since our requirements have changed, and we will not be needing this in the near future.
If anyone else wants to pick this up, the maintainers would be happy to review PRs or provide assistance.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
