k8s-gitops
k8s-gitops copied to clipboard
billimek
explore kubevirt
Goal:
remove need for hypervisor layer (proxmox). Currently this is serving three purposes outside of kubernetes:
- 'home' VM with a linux shell for development and other linux home type activities
- 'NAS' server VM, serving up zfs tank NFS/samba share to the rest of the network
- Blue Iris - windows server VM running DVR software; requires GPU
The first two should be easy solves (the second can probably be done as a deployment in conjunction with rook probably). a windows kubevirt with GPU offload is a stretch goal as I think kubevirt only supports nvidia GPU passthrough at the moment, and my use-case is intel iGPU.
To look at:
- https://kubevirt.io/
First pass looking at this wasn't so awesome:
- Running kubevirt in a k8s node which is, itself, a VM doesn't allow native hardware-based VM instructions from the CPU, even though the CPU is being passed directly and not virtualized.
- YOLOing through the quickstart docs to get a regular generic VM started didn't work, so more care and attention will be required to give this a good test
- Uninstalling kubevirt left behind overly-aggressive webhooks which broke flux and velero. This wasn't great, and required some additional cleanup commands to properly uninstall kubevirt.
What do you mean by "doesn't allow native hardware-based VM instructions". I believe in x86 there are like 16 instructions from ring 0 that cannot be just passed through. Anything specific you have in mind?
I went through the kubevirt installation process recently and documented it all. If you have any specific questions I might be able to help out.
Hi @mathias-ewald, It's been a long while but thinking back I think it was related to instructions required to do some tensorflow processing. From what I recall, when running a workload on a VM with an emulated CPU, there were certain instructions (I think AVX) not present with an emulated CPU that tensorflow requires. This may be what I was referring to but not positive anymore.
Normally, that is not the case, but hypervisors can mask certain instruction sets when configured to. Usually, this is only done to maintain live migration compatiblity (migrate a VM from on host to another). Imagine you start a VM on a CPU with instructions A, B and C and then live migrate it to a host with only A and B. Won't work, but if you mask C on that first host, it'll do. Assuming you're using KubeVirt with QEMU, have a look at the "-cpu" parameter to qemu. Must be set to "host" or something for full passthrough.