community
community copied to clipboard
kubevirt
Generalize virtualization stack to different libvirt hypervisor-drivers
This KubeVirt design proposal discusses how KubeVirt can be used to create libvirt virtual machines that are backed by diverse hypervisor drivers, such as QEMU/KVM, Xen, VirtualBox, etc. The aim of this proposal is to enumerate the design and implementation choices for enabling this multi-hypervisor support in KubeVirt.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Checklist
This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR. Approvers are expected to review this list.
- [ ] Design: A design document was considered and is present (link) or not required
- [ ] PR: The PR description is expressive enough and will help future contributors
- [ ] Code: Write code that humans can understand and Keep it simple
- [ ] Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
- [ ] Upgrade: Impact of this change on upgrade flows was considered and addressed if required
- [ ] Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
- [ ] Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
- [ ] Community: Announcement to kubevirt-dev was considered
Release note:
Hi @harshitgupta1337. Thanks for your PR.
PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign fabiand for approval. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
@harshitgupta1337 I think what we need here it is an infrastructure for hypervisor plugins. Putting everything into kubevirt code would make the code base very large and hard to maintain. Do you think it would be possible to change the goal of this proposal and try to develop a plugin instead. Cloud Hypervisor could be the first hypervisor plugin.
I think we need also a list of features supported by each specific hypervisor.
From the old proposal for Cloud Hypervisor integration, not all the features supported by QEMu are available in CH. We should have here also a similar section with the features as in the old proposal. Then, these options should be advertised in the new CRD if they are supported or not.
@harshitgupta1337 I think what we need here it is an infrastructure for hypervisor plugins. Putting everything into kubevirt code would make the code base very large and hard to maintain. Do you think it would be possible to change the goal of this proposal and try to develop a plugin instead. Cloud Hypervisor could be the first hypervisor plugin.
@alicefr You're referring to this feature in Golang, isn' it ? https://pkg.go.dev/plugin Then we would need 2 plugins, right? One for qemu for backwards compatibility and a new one for cloud-hypervisor?
@harshitgupta1337 I think what we need here it is an infrastructure for hypervisor plugins. Putting everything into kubevirt code would make the code base very large and hard to maintain. Do you think it would be possible to change the goal of this proposal and try to develop a plugin instead. Cloud Hypervisor could be the first hypervisor plugin.
@alicefr You're referring to this feature in Golang, isn' it ? https://pkg.go.dev/plugin Then we would need 2 plugins, right? One for qemu for backwards compatibility and a new one for cloud-hypervisor?
No, I'm referring more to an API where we can plug different hypervisors, like CRI for the container runtimes
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale
/cc I haven't looked at the proposal yet, but wanted to point out that this proposal reminds me of a similar design proposal from the past: https://github.com/kubevirt/community/pull/184.
This proposal is not focused specifically on cloud-hypervisor, which I think is a step forward. I think that reading the discussion there will be interesting and valuable, especially this comment. To summarize:
So we'd need a similar interface to CRI, but for hypervisor runtimes for kubevirt.
I think that it will definitely not be easy to design and implement something like that, especially since currently Kubevirt heavily relies on libvirt-specific features/APIs. Saying that, if you're willing to invest the effort to achieve it, I think it would be valuable and I will be happy to promote it.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale
@iholder101 Yes I am working on addressing the suggestions mentioned earlier. I should be able to come up with an updated design by the end of this month.
@iholder101 Yes I am working on addressing the suggestions mentioned earlier. I should be able to come up with an updated design by the end of this month.
Great to hear that! Good luck, let me know if I can help in any way.
/remove-lifecycle stale
@harshitgupta1337 I think what we need here it is an infrastructure for hypervisor plugins. Putting everything into kubevirt code would make the code base very large and hard to maintain. Do you think it would be possible to change the goal of this proposal and try to develop a plugin instead. Cloud Hypervisor could be the first hypervisor plugin.
@alicefr You're referring to this feature in Golang, isn' it ? https://pkg.go.dev/plugin Then we would need 2 plugins, right? One for qemu for backwards compatibility and a new one for cloud-hypervisor?
No, I'm referring more to an API where we can plug different hypervisors, like CRI for the container runtimes
Hi @alicefr I am curious why a CRI-like interface is necessary in this scenario, given that libvirt provides a uniform abstraction for creating VMs?
@harshitgupta1337 In the code (virt-handler's one to be specific) you can find out that we directly configure qemu as well. So as such Libvirt does support multiple "runtimes" but since the libvirt is not privilege in Kubevirt, it can't do everything.
Open question: How to manage the features that are supported on one Virt Stack and not on another?
Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:
- 2ea83ae Add design proposal for KubeVirt generalize virt stack
- 61244c9 Add design text. Diagram pending
- 1e50a6f Add Implementation Phases
- bb7ad21 Add KubeVirt VMI flow
- 7fc0c24 Update
- d1036a5 Update goals
- 4b9a5ce Add interface code
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.
Hey @everyone, I have updated the design proposal with a couple of interfaces that I have added in my PoC that expose necessary functions for supporting both qemu-kvm and cloud-hypervisor. @vladikr