Support Qemu accelerated podman machine on Windows hosts

[arixmkii]

/kind feature

Description

Support Qemu accelerated podman machine on Windows hosts.

• gvproxy supports windows in its latest releases https://github.com/containers/gvisor-tap-vsock/releases/tag/v0.3.0
• There are Qemu windows builds, which supports Hyper-V based acceleration https://qemu.weilnetz.de/w64/ so, -machine q35,accel=whpx:tcg should perform snappy on Hyper-V enabled hosts.

I'm pretty new to podman machine, so, I might be not aware of other external components needed to run podman machine.

I saw that there is support for running podman machine in WSL2 being merged to 4.0.0 RC2. Why this idea still makes sense.

• It will be Fedora CoreOS based;
• It will be fully featured Linux VM;
• It will more precisely match behavior of the podman machine on MacOS;
• It will allow running at least some arm64 images with qemu syscalls for guest architecture on windows hosts.

The downside from WSL2 option would be less efficient host resource management, but in some cases it is not of the concern.

What is missing right now is the command line support for choosing between multiple available vendors for podman machine on a single platform, which I don't think is a thing already (at least from my studying of the source tree).

[rhatdan]

I am fine with this, but we need community to work on it. If you want to implement this and it is simply a CLI change, I would be all for it.

[afbjorklund]

@arixmkii

The main problem was the lack of unix sockets, for the Windows build. Then there were a lot of other "little things", to fix.

Here some old code, if you want to continue with it, https://github.com/afbjorklund/podman/commits/machine-windows

Things like missing "xzcat.exe" and missing $HOME, and so on and so forth.

The official Podman support would be for WSL2, so it (Windows) was abandoned.

• https://github.com/containers/podman/pull/12503

Like you say, qemu itself should support windows and co-exist with Hyper-V.

[arixmkii]

@afbjorklund Thank you for your input! I have interest to at least try to make it work, but I need to study the code in more details, it will take time.

The official Podman support would be for WSL2

Feel free to close this issue then with this resolution, it could be reopened/recreated later if there is some actual work to base it on.

[afbjorklund]

There was windows support in podman-machine (v1) but it mostly used VirtualBox so the QEMU driver was bare-bones...

It should be possible to use "fifo files" instead of "unix sockets", but it will take some adaption of the go code and qemu flags.

[albertdb]

This would be really nice to have. Currently the only alternative, where WSL2 is not feasible, is using Minikube, which at the current version includes Podman v2.

[afbjorklund]

You could also DIY, using something like Vagrant for the provisioning. But otherwise I think "everyone" is using WSL2 now.

https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/about/supported-guest-os

[arixmkii]

The main problem was the lack of unix sockets

Go has built in support for unix sockets on windows https://go-review.googlesource.com/c/go/+/125456/ which I expect could work just fine for gvproxy and podman binaries.

Mingw added afunix.h last summer (pretty fresh and probably will need to build it from sources to get it into toolchain) https://github.com/mingw-w64/mingw-w64/blob/43e87a27fdb97ae562f7c1e2017c8ce58fef9ee1/mingw-w64-headers/include/afunix.h I believe then Qemu sources could be patched to allow unix sockets in windows build at least behind conditional switch.

Windows support lacks DATAGRAM for unix sockets, but I'm not ready to say if this will be a blocker or not.

missing "xzcat.exe"

Download latest windows build of xzutils https://tukaani.org/xz/ and copy or remane xz.exe to xzcat.exe and it will behave as xzcat, because it checks the command line first argument to choose the behavior. More natural with symlinks, but this way it will work on windows - checked SHA256 of processed qcow2.xz on Win and MacOS hosts.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[arixmkii]

There is a patch submitted to QEMU for enabling builds with af_unix support on Windows https://lists.gnu.org/archive/html/qemu-devel/2022-08/msg00221.html

[arixmkii]

This also looks useful https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg04098.html as the functionality of file descriptors for unix sockets on windows seems to be limited in Go lang/runtime (probably platform limitations).

It will be needed to replace gvproxy qemu_wrapper pattern from podman.

[arixmkii]

Current progress:

Nginx arm version running on 64 bit Win 11. Works for core and root users. Root version allows to forward privileged ports to host (80 -> 80 for nginx).

Using

• modified qemu (this branch https://github.com/arixmkii/qemu/tree/win_sock which added changes mentioned here https://github.com/containers/podman/issues/13006#issuecomment-1203760996)
• modified gvproxy (submitted for upstream https://github.com/containers/gvisor-tap-vsock/pull/140)
• modified podman

Had to switch it to cni as netavark doesn't work for me for some reason. Don't have enough knowledge to troubleshoot netavark issues now, but will look into it later.

[arixmkii]

Noticed that there is some work for Apple virtualization for Apple Silicon in the source tree, but didn't manage to find any code that allows switching between virtualization providers (Apple vs Qemu) for this platform. Wanted to use something ready for my code, but it seems that this part is not ready yet.

[rhatdan]

Yes it is planned for the future.

[gbraad]

@rhatdan Please note that this is not something we can easily deliver as an MSI install, as this relies on a third-party build of Qemu. The whpx support is not a standard build flag, and just like WSL2 relies on Hyper-V. Why not add Hyper-V support instead?

@arixmkii we are currently looking at vfkit as was added to crc. @cfergeau and @baude are working on this.

[arixmkii]

as this relies on a third-party build of Qemu

This is temporary. It is built from master with only single patchset applied (where 1 of 4 patches is already in master, so, I believe it is on its way in).

The whpx support is not a standard build flag

It looks like it is default at least for msys/cygwin builds. I didn't try cross compile. And it is enabled in windows builds advertised on Qemu home page.

WSL2 relies on Hyper-V

It should be possible to use Intel HAXM as well. I have plans for this experiment.

Why not add Hyper-V support instead?

Hyper-V is already supported in a form of WSL2, but it relies on modified kernel and can't run coreos due to some limitations (systemd? cgroups?).
Raw Hyper-V is an interesting topic to investigate, but the point here was to bring as similar as possible "podman machine" experience to all major platforms - Windows, MacOS, Linux (I haven't tested Qemu machine on Linux, but from code perspective it is there).

@gbraad Thank you for the heads-up on vfkit! Nice to see something friendlier than vf API. My interest here was if there is already a CLI support for having multiple providers for the same target and it seems there is nothing yet (not a blocker right now).

[gbraad]

We would have to maintain our own builds of Qemu, just like we do for macos. Else it would be hard to provide 'support' or any form of consistent behaviour. For macos we keep a build in the containers organization. Perhaps same needed for Windows

[gbraad]

HyperV and WSL2 work quite differently. For WSL2 we use a prepared image that is in the podman-WSL2-Fedora subproject, but this is a modified container image. For HyperV we would have a full VM and more control over the network stack and usage, like we do for CRC. More differences exist, like 9P stack, shared network stack, etc

Note: Qemu on Windows, even with WHPX, performs slower.

Note 2: we have used haxm too in tests, but this conflicts with other hypervisors (like having WSL2 enabled). In the end it did not feel much better

[arixmkii]

We would have to maintain our own builds of Qemu

Oh. Didn't know that. I only used homebrew versions. Now I realized that it is formulae and they are doing rebuilds. Didn't check the official installer package.

I believe Google/Android studio is doing something like this with their Qemu based emulator.

HyperV and WSL2 work quite differently.

No doubt here. Added this comment for the completeness of the context in our discussion.

Qemu on Windows, even with WHPX, performs slower

Haven't compared them side by side, but sounds logical. I don't consider this as production load runner, but development assistant workflow mostly. Also, Qemu allows to run and build arm64 containers on amd64 (not for every workload of course).

[arixmkii]

Another nice to have patch set for qemu - 9pfs/virtf windows hosts https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg04075.html

[arixmkii]

Offtopic a bit, but would appreciate any pointers, where I can find more details for netavark and how to troubleshoot it. 😅

[arixmkii]

Collected my changes into single commit https://github.com/arixmkii/podman/commit/cc5c8d990c4088a92e5db79308d7bffd2ffeb2f6

Will try to upstream at least some parts of it, because some of the changes are not bound to this issue.

[mheon]

@arixmkii What kind of details on netavark? Documentation is a little minimal, but that's something we'd like to fix.

[arixmkii]

@mheon I don't know yet 😓 I have these for now Last debug message from netavark

[DEBUG netavark::network::core] Container veth mac:

And then teardown procedure of podman run command resulting in this output

Error: netavark: : EOF

EOF I guess is connection interruption on app crash.

I will compare it with the output on MacOS (where it works just fine) and then will look at the sources and workflow of netavark for the potential culprit.

For now I'm just very satisfied that CNI worked for my experiment 🙂

Update: Discovered that trace level gives out the full command, this is something to play with.

[arixmkii]

Found in journald logs that netavark failed with crash dump due to missing CPU extensions in VM. Switched Qemu CPU to -cpu Skylake-Server-v5 and it resolved the issue with netavark.

Need to figure out why -cpu host is not available on windows. Tried -cpu max, but VM crashed - probably due to my machine specifics.

So, I would say that now I have working Qemu machine on Windows. Plan to create v2 branch, when currently created PRs are merged and hopefully some more registered issues resolved (to make changes even more minimal).

Update: -cpu Skylake-Client-v4 also works. Probably this one is a safer option.

[n1hility]

Qemu on Windows, even with WHPX, performs slower

Haven't compared them side by side, but sounds logical. I don't consider this as production load runner, but development assistant workflow mostly. Also, Qemu allows to run and build arm64 containers on amd64 (not for every workload of course).

In this case, are you referring to qemu-user-static to run mixed arch containers? If so we can support that on any virt backend technology. IIRC it's already shipped in latest FCOS.

Or do you mean using a full system non-virt emulation (e.g. TCG) with qemu? IMO former is better than the latter.

While QEMU support on Win is interesting, I think the ideal long-term solution for a pure virt alternative on Windows is likely Hyper-V. Primarily for the reasons that @gbraad mentioned: it's included out of the box, and it is already well integrated with the host OS.

[arixmkii]

In this case, are you referring to qemu-user-static to run mixed arch containers? If so we can support that on any virt backend technology. IIRC it's already shipped in latest FCOS.

That's great. So any hypervizor running full FCOS will do the trick? Is this correct?

full system non-virt emulation (e.g. TCG)

Having this option (with instruction provided how to acquire needed qcow, adjust command line in json file, etc) might have occasional uses at least with buildah (sometimes foreign architecture just doesn't work for specific apps and with buildah performance can be traded (sometimes)). But this is from "advanced" topics, not something, what should be aimed for (at least for now).

While QEMU support on Win is interesting

I got the experience on Win I expected. And the changeset is not that big, so, I will try to upstream as many parts as possible and potentially setup a separate repo, where I can do rebuilds for Qemu with patches, podman with Qemu support on windows, etc.

While QEMU support on Win is interesting

I though that it makes sense, because there is a possibility to get it working with only minimal changes. Also, it looks like Qemu will still be used for podman machine on Linux and would be kept around at least until Intel Macs are supported (I don't know if there are plans to remove the Darwin support after vfkit version is ready for Apple Silicon Macs).

I think the ideal long-term solution for a pure virt alternative on Windows is likely Hyper-V.

Sounds very reasonable, but this is significantly different effort.