[Windows Please]

[jhay06]

Description

Hi , This is such a great work , hoping for the Windows version

WSL is good but very slow in most cases , i prefer to have this in windows , :) hope that this will be available soon.

Thanks

[jandubois]

WSL is good but very slow in most cases

I don't see how we could improve on the performance of WSL and Hyper-V on Windows. So unless somebody else has ideas, and the ability to implement them, this is unlikely to happen.

[afbjorklund]

I think you can use the QEMU binaries with the WHPX acceleration, along with any available ISO, to see the "baseline" performance. If that is not enough, I guess this is more a request for running Windows containers (like Docker) ?

https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/

I would be OK with being able to run VMs the same way on Windows, that is available on Mac and Linux today (i.e. QEMU). We had this with docker-machine and podman-machine, so it should be "possible" also for containerd-machine (lima)

But I don't run Windows myself, and when I do it is with something like MSYS

[afbjorklund]

There are some GOOS=windows compilation issues on master, but those should be easy to fix:

# github.com/lima-vm/lima/pkg/lockutil
pkg/lockutil/lockutil.go:34:27: undefined: unix.LOCK_EX
pkg/lockutil/lockutil.go:38:28: undefined: unix.LOCK_UN
pkg/lockutil/lockutil.go:48:10: undefined: unix.Flock
pkg/lockutil/lockutil.go:49:27: undefined: unix.EINTR
note: module requires Go 1.18
# github.com/lima-vm/lima/pkg/networks
pkg/networks/validate.go:76:25: undefined: syscall.Stat_t
note: module requires Go 1.18

The "lockutil" are just missing some nerdctl code available. The syscall needs wrapping...

https://github.com/containerd/nerdctl/tree/master/pkg/lockutil

---

EDIT: added:

• #911
• #912

make GOOS=windows _output/bin/limactl: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows

[afbjorklund]

Something like: (see MSYS2, and https://www.alpinelinux.org/downloads/)

$ /c/Program\ Files/qemu/qemu-system-x86_64 -m 512 -smp 1 \
  -accel whpx,kernel-irqchip=off -cdrom alpine-virt-3.16.0-x86_64.iso
Windows Hypervisor Platform accelerator is operational

The display looks broken (no input), but -serial stdio almost works.

ISOLINUX 6.04 6.04-pre1  Copyright (C) 1994-2015 H. Peter Anvin et al
boot:


   OpenRC 0.44.10 is starting up Linux 5.15.41-0-virt (x86_64)

 * /proc is already mounted
 * Mounting /run ... * /run/openrc: creating directory
 * /run/lock: creating directory
 * /run/lock: correcting owner
 * Caching service dependencies ... [ ok ]
 * Remounting devtmpfs on /dev ... [ ok ]
 * Mounting /dev/mqueue ... [ ok ]
 * Mounting modloop  ... * Verifying modloop
 [ ok ]
 * Mounting security filesystem ... [ ok ]
 * Mounting debug filesystem ... [ ok ]
 * Mounting persistent storage (pstore) filesystem ... [ ok ]
 * Starting busybox mdev ... [ ok ]
 * Loading hardware drivers ... [ ok ]
 * Loading modules ... [ ok ]
 * Setting system clock using the hardware clock [UTC] ... [ ok ]
 * Checking local filesystems  ... [ ok ]
 * Remounting filesystems ... [ ok ]
 * Mounting local filesystems ... [ ok ]
 * Configuring kernel parameters ... [ ok ]
 * Migrating /var/lock to /run/lock ... [ ok ]
 * Creating user login records ... [ ok ]
 * Cleaning /tmp directory ... [ ok ]
 * Setting hostname ... [ ok ]
 * Starting busybox syslog ... [ ok ]
 * Starting firstboot ... [ ok ]

Welcome to Alpine Linux 3.16
Kernel 5.15.41-0-virt on an x86_64 (/dev/ttyS0)

localhost login: root
root
Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org/>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.

localhost:~#

---

EDIT: The kernel-irqchip thing was a workaround for a startup error:

whpx: injection failed, MSI (0, 0) delivery: 0, dest_mode: 0, trigger mode: 0, vector: 0, lost (c0350005)

And with "almost works", I mean this console has some weird issues:

localhost:~# apk add containerd
pk add containerd
-ash: pk: not found

EDIT: -display sdl works (better than "gtk")

Here the console interaction works better.

[afbjorklund]

So lima "works", and qemu "works". Left to do is making them work together, and add some documentation. 😃

Accelerator: https://docs.microsoft.com/en-us/virtualization/api/hypervisor-platform/hypervisor-platform (WHPX)

WSL is good but very slow in most cases , i prefer to have this in windows , :)

The main difference between WSL2 and Lima, is that lima uses a new virtual machine for each instance... With the Windows Subsystem for Linux, all the system containers share the same VM kernel (like in LXC)

It is possible to start one Linux distribution (like Alpine), and then start system containers for Ubuntu or whatever. Then the experience should be similar, same goes with sharing files - if opting in to use 9p (same as WSL uses)

[afbjorklund]

Almost got it to run, final hurdle is converting paths for qemu (dos, argh) and scripts for ssh (don't ask)

"[hostagent] qemu[stderr]: C:\\Program Files\\qemu\\qemu-system-x86_64.exe: cannot create PID file: Failed to create PID file"
"[hostagent] stdout=\"\", stderr=\"command-line line 0: invalid quotes\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"command-line line 0: invalid quotes\\r\\n\": exit status 255"

Fixes (PRs):

• lima compiles for GOOS=windows, cross-compiled on linux
• unittests runs for GOOS=windows, using wine64 on linux

Verified:

• regular limactl.exe operations (download, etc) works ok on Windows 10
• starting virtual machine with hardware acceleration works on Windows 10

Fallbacks:

• fallback to user "lima" using existing code, due to DOMAIN\user
• use id -u and id -g where available, otherwise fallback uid gid
• add home directory to the LimaUser, instead of using it "raw"
• use cygpath $HOME where available, otherwise just use "filepath"
• use windows paths (filepath) for host home and unix paths (path) for guest home
  • https://github.com/lima-vm/lima/pull/954

Workarounds:

• use tcp sockets instead of unix sockets, for hostagent/guestagent (needs tls)
• use named pipes instead of unix sockets, for qemu communication (needs patch)
• use unix paths for iso9660, since go-embed hardcodes slashes in filenames
  • https://github.com/lima-vm/lima/pull/953

User needs to add qemu, and regular tools - either MSYS2 or Git for Windows (MinGW) would work...

It's all normal programs, so it would be possible to install qemu-system-x86_64.exe and ssh.exe etc. It does not require a Unix environment (like Cygwin) or other emulator, besides the regular QEMU (and Lima).

Using the "whpx" accelerator requires Windows with Hyper-V (Pro?), falling back to "haxm" would be possible.

---

Will make a PR for the fallbacks, but the rest needs a design decision - or to wait for AF_UNIX support ?

At this point it is just a proof-of-concept or technical demo, users are still recommended to use WSL2.

Note: this does not improve the performance (with Hyper-V), but it should be on par with the Mac version ?

I assume that all developers will be using Unix, and will not set up anything for PowerShell or DOS etc.

[arixmkii]

AF_UNIX support ?

@afbjorklund are you aware if there is any activity enabling AF_UNIX for windows builds on qemu side? This could benefit other projects as well. Like podman providing podman machine with MacOS like behavior instead as an alternative to WSL2 option.

[afbjorklund]

Sorry, I don't know anything about it. The information I stumbled upon so far looked more like "gross hacks" than anything else.

https://cygwin.com/pipermail/cygwin/2020-June/245088.html

https://stackoverflow.com/questions/23086038/what-mechanism-is-used-by-msys-cygwin-to-emulate-unix-domain-sockets

I will assume that Unix sockets are unavailable on Windows

[afbjorklund]

Afaik, Podman only uses Unix sockets for legacy (pre 18.09) Docker clients ? The other clients use SSH directly

[arixmkii]

Podman machine uses unix socket for qmp at least

-qmp unix://var/folders/<redacted>/T/podman/qmp_podman-machine-default.sock,server=on,wait=off 

And looks like for virtio-serial device

-device virtio-serial -chardev socket,path=/var/folders/<redacted>/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=podman-machine-default_ready 

These are extracts from podman machine start command line running on MacOS.

[afbjorklund]

Oh, I thought you meant for the podman connection... (Formerly known as CONTAINER_HOST or PODMAN_USER/PODMAN_HOST/PODMAN_PORT)

[arixmkii]

No. I was talking about podman machine command and framework specifically. Having AF_UNIX in QEMU windows build could reduce the amount of platform specific branches to implement QEMU backed podman machine command for modern Windows versions.

[afbjorklund]

For the PoC, I just used -chardev pipe (and mkfifo) for the qemu control.

       -chardev pipe,id=id,path=path
           Create a two-way connection to the guest. The behaviour differs slightly between Windows hosts and other hosts:

           On Windows, a single duplex pipe will be created at \\.pipe\path.

           On other hosts, 2 pipes will be created called path.in and path.out. Data written to path.in will be received by the guest. Data written by the guest can be read
           from path.out. QEMU will not create these fifos, and requires them to be present.

           path forms part of the pipe path as described above. path is required.

Didn't bother creating a qmp Monitor for Unix though, "left as an exercise"

[afbjorklund]

But otherwise, I would be happy enough if exec.Command actually worked (with filepath)

Note that the examples in this package assume a Unix system. They may not run on Windows, and they do not run in the Go Playground used by golang.org and godoc.org.

https://pkg.go.dev/os/exec#Command

On Windows, processes receive the whole command line as a single string and do their own parsing.

[afbjorklund]

Turns out that qemu doesn't start up correctly with pipe chardev. Switch them to null, and it works.

The WHPX accelerator is not compatible with -cpu max, so that needs a special case (like -cpu host)

• #931

Using Wine is too unstable to do anything but run unit tests, even with -accel tcg there are random failures.

The path issues were related to that os.UserHomeDir value is not compatible with exec.Command...

In case your home directory is C:\Users\AndersBjörklund or something, it fails to encode it properly.

This affects the default $LIMA_HOME and ~/.ssh, so probably needs some $HOME workaround/fallback <sigh>.

[afbjorklund]

More random whpx failures:

{"level":"debug","msg":"qemu[stdout]: Windows Hypervisor Platform accelerator is operational","time":"2022-06-29T17:02:37+02:00"}
{"level":"debug","msg":"qemu[stderr]: C:\\Program Files\\qemu\\qemu-system-x86_64.exe: WHPX: Failed to emulate MMIO access with EmulatorReturnStatus
: 2","time":"2022-06-29T17:02:37+02:00"}
{"level":"debug","msg":"qemu[stderr]: C:\\Program Files\\qemu\\qemu-system-x86_64.exe: WHPX: Failed to exec a virtual processor","time":"2022-06-29T
17:02:37+02:00"}
{"error":"exit status 3","level":"info","msg":"QEMU has exited","time":"2022-06-29T17:02:37+02:00"}

Come and go, mysteriously...

It seems to mostly affect the default (ubuntu) image, even though only the ISO / URL changes ?.

Not the alpine (alpine-lima) image, but unfortunately it does not include any nerdctl/containerd.

[afbjorklund]

Unfortunately, the terminal detection and signal handling is all messed up.

time="2022-06-29T17:25:44+02:00" level=info msg="Terminal is not available, proceeding without opening an editor"

• #936

---

If you terminate the limactl shell, then the limactl start kills the qemu.

{"level":"info","msg":"Received SIGINT, shutting down the host agent","time":"2022-06-29T17:23:14+02:00"}

[afbjorklund]

In theory, this would be the way to fix the home:

\\?\C:\Users\AndersBjörklund (UNC)

In practice, this is the only workaround that works:

C:\Users\ANDERS~1 (DOS)

---

Probably want to flip these internal paths back to regular again, before displaying them to the user ?

Ironically, this seems to be done using filepath.ResolveSymlinks (which currently breaks LimaDir)

EDIT: Added PR, instead of hardcoded string:

• https://github.com/lima-vm/lima/pull/955

[afbjorklund]

Basic operation on Windows 10, when using Alpine with a custom containerd + nerdctl installation.

limactl start template://alpine

• lima sudo apk add containerd cni-plugins
• lima sudo /etc/init.d/containerd start
• https://github.com/containerd/nerdctl/releases

$ limactl ls
NAME      STATUS     SSH                ARCH      CPUS    MEMORY    DISK      DIR
alpine    Running    127.0.0.1:51129    x86_64    4       4GiB      100GiB    C:\Users\ANDERS~1\.lima\alpine

$ uname
MINGW64_NT-10.0-19044

$ lima uname
Linux

$ lima sudo nerdctl version
Client:
 Version:       v0.21.0
 OS/Arch:       linux/amd64
 Git commit:    9ddf5226eabcbb7b4b43987f3b0f8d53d86d3bca

Server:
 containerd:
  Version:      v1.6.6
  GitCommit:    10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1

Note: no mounts, until the host/guest path situation is sorted out

DEBU[0002] the host home does not seem mounted, so the guest shell will have a different cwd

Note: no virtfs on windows, which means no 9p only sshfs mounts

ERROR: Feature virtfs cannot be enabled: virtio-9p (virtfs) requires Linux or macOS

---

Ubuntu template is still broken, MSYS2 terminal is still broken ("invalid quotes")

hostagent/useragent uses insecure ports, and qmp/serial sockets are disabled...

[afbjorklund]

This is the EFI bug, turns out alpine still uses BIOS:

https://gitlab.com/qemu-project/qemu/-/issues/513

Seems like a workaround is to use -bios instead ?

EDIT: Indeed, that was it (with a custom OVMF.fd)

So now both images are working OK, with WHPX.

---

• https://github.com/lima-vm/lima/pull/982

[afbjorklund]

Fixed the quoting issues for MSYS, so now all three consoles should work (with lima)

1. MSYS2 (msys64 subsystem)
2. MinGW64 (Git for Windows)
3. Command Prompt (cmd.exe)

Will push "port" and "pipe" up as drafts, and rebase and clear up the home directory...

• port: use tcp sockets instead of unix sockets, for hostagent/guestagent
• pipe: use named pipes instead of unix sockets, for qemu communication
• add better handling of the external OVMF_CODE.fd, unlike the internal BIOS

[afbjorklund]

Typical output:

MSYS2

MinGW64

cmd.exe

[AkihiroSuda]

Thanks a lot @afbjorklund

port: use tcp sockets instead of unix sockets, for hostagent/guestagent

This is fine until ssh.exe supports UNIX sockets, but this TCP socket has to be protected with mTLS to avoid potential attacks from malicious web sites via WebSockets.

[afbjorklund]

This is fine until ssh.exe supports UNIX sockets, but this TCP socket has to be protected with mTLS to avoid potential attacks from malicious web sites via WebSockets.

I know, that is why I left it in draft. It's the same status as Docker's port 2375 - ok for testing development, but needs port 2376 for deployment production. Same thing with the named pipes unfortunately, currently it is using "null" instead of "pipe" in qemu.

       -chardev pipe,id=%s,path=%s

       -chardev socket,id=%s,path=%s,server=on,wait=off

Anyway, I will put the code up there for reading - hopefully there is some reasonable implementation to add tls to it (?), and hopefully there is some easy fix / patch to qemu for windows to allow it to still boot even when given the pipe option.

• https://github.com/lima-vm/lima/pull/980

• https://github.com/lima-vm/lima/pull/981

---

Investing some weird panic with the dns server as well, commented it out - but need to find out why it won't start...

                logrus.Debugf("Start %v server listening on: %v", network, addr)
                if e := s.ListenAndServe(); e != nil {
                        panic(e)
                }

So it remains in the "proof of concept" status, reason for pushing it is so that any Windows developer can help out.

[geoman2]

Hi, found this repo recently and would like to know if the win10 version can be installed and what features are missing?

[afbjorklund]

Everything that is needed to build for windows should be present in the main branch, but running it requires patches...

1. Allow the hostagent/guestagent to run (#980)
2. Allow QEMU with qmp and serial to run (#981)
3. More work needed on path conversion
4. Something wrong with the DNS server

The main system requirements would be Windows 10 (or 11), with Hyper-V support. You also need QEMU with WHPX.

• https://qemu.weilnetz.de/w64/2022/qemu-w64-setup-20220419.exe

You can test your VM environment, using something like https://www.alpinelinux.org/ (qemu-system-x86_64 -accel whpx)

• https://dl-cdn.alpinelinux.org/alpine/v3.16/releases/x86_64/alpine-standard-3.16.1-x86_64.iso

---

For developing, I would recommend something like MSYS2 (C:\msys64) since it has all tools... And go, from https://go.dev/

Before it is generally available, the tcp socket needs to be protected (using a similar ca/cert/key setup as with docker tls) And need some better fix on how to control qemu, with the pipes and processes. Now it can be hanging in the background.

There is probably more, so it would be great with some feedback and some more testing... Aim is to be "same as other OS"

Here is what it looks like on Ubuntu:

qemu-system-x86_64 -accel kvm -m 512 -cdrom alpine-standard-3.16.1-x86_64.iso -display default -nic user

It should look very similar on Windows.

[geoman2]

Thank you

[AkihiroSuda]

This patch might be useful: [PATCH v4 0/4] Enable unix socket support on Windows https://lists.gnu.org/archive/html/qemu-devel/2022-08/msg00221.html

[arixmkii]

This patch might be useful: [PATCH v4 0/4] Enable unix socket support on Windows

I have Qemu rebuilds (now at 7.1.0-rc4) with this one and additionally one more [PATCH v7 00/14] qapi: net: add unix socket type support to netdev backe https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg04098.html

I managed to use this build to run podman machine with QEMU without opening tcp sockets and not using pipes. Might try it out with Lima as well.

There is another nice to have patch [RFC PATCH 0/4] 9pfs: Add 9pfs support for Windows host https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00983.html

But this doesn't play well with recent changes in 9pfs. Hope that authors will resubmit it after 7.1.0 release.

[arixmkii]

I published QEMU 7.2.0 build with added 9pfs support on Windows as part of https://github.com/arixmkii/qcw/releases/tag/v0.0.7

Hope to find some time during holidays to experiment with lima on Windows machine.