QEMU / KVM Processors

[the-wondersmith]

I've got my homelab set up and working beautifully as a 3-node Pi4/pimox7 cluster (can't thank you enough for putting the project together).

I'd previously had an single-node Proxmox instance running on an ancient HP ProLiant (before catastrophic failure) which made cross-os software development work a breeze. I have been trying to get back to some semblance of the functionality I previously had.

It seems like what's preventing me from spinning up a Win10 VM using kvm64 as the CPU is QEMU missing several of ProMox's usual CPU types. It also looks like the version of QEMU pimox uses isn't the same as what you'd get if you ran apt install -y qemu-system-arm64.

tl;dr I'd really like to spin up an x86 or x86_64 VM, but pimox doesn't seem able to at the moment. What can I do to help resolve that? (For clarity, software development is my day job. When I say help, I mean anything up to and including source-level patches.)

[the-wondersmith]

Well, I'm going to assume that the ability to compile Proxmox source(s) is going to factor into it which led me to the Proxmox git repos which don't necessarily have a convenient "pull down all the repos" sorta deal. I ended up creating a little helper script to install all the Proxmox dev dependencies and then pull down all the Proxmox repos. I've attached the script to this post in case it's helpful for anyone else who is following along at home.

I'll try and keep this issue updated with any progress I make.

[aanon4]

Are you trying to spin up an x86 VM on a Pi? It can work .. its very very very very slow though.

[the-wondersmith]

Are you trying to spin up an x86 VM on a Pi? It can work .. its very very very very slow though.

I am, yes. I'm fine with it being slow (at least as a starting point), I'm more looking at it as an opportunity to more fully bring Proxmox to the ARM platform in general (and RPi in particular).

[aanon4]

I quickly tried this on one of my ARM nodes. Didn't do the install but got as far as it booting up the installer. Seems fine but it's been a while since I tried this in a serious way. Does something like this work for you? Now .. I've never tried Windows.

arch: x86_64 balloon: 0 boot: order=scsi2 cores: 1 memory: 2048 name: test-x86-on-arm net0: virtio=F2:2F:E1:0B:CF:CC,bridge=vmbr0,tag=51 numa: 0 ostype: l26 scsi0: local:113/vm-113-disk-0.qcow2,size=2G scsi2: isos:iso/debian-9.4.0-amd64-netinst.iso,media=cdrom,size=291M scsihw: virtio-scsi-pci smbios1: uuid=fa80b004-9ae9-4c00-a291-71f115d17710 sockets: 1

[the-wondersmith]

@aanon4 Apologies for the delay in getting back to you. I've been mucking about with things while trying to get this to work and of course broke my cluster.

I'm getting it all set back up and will report in as soon as I'm able.

[Sysadminfromhell]

Can we get Windows somehow to work? If I try and run through the Wizard it cant start the VM:

kvm: -no-hpet: Option not supported for this target TASK ERROR: start failed: QEMU exited with code 1

VM-Config:

EDIT:

Nevermind, could fix it with adding arch: x86_64 to the config 100.

EDIT2: Installation works but cant boot into it. Removed the VM for now.

[the-wondersmith]

@Sysadminfromhell I've been fiddling with my three-node cluster since this thread was opened and have not once yet been able to so much as get PiMox to install properly (again). Something goes wrong literally every single time, most often the ZFS module.

It became so frustrating that I eventually just wiped all three nodes and burned fresh Ubuntu Server 21.10 arm64 images onto the drives and manually set up a three-node LXD cluster. It's way less convenient than Proxmox, but so far I can confirm that it at least seems stable.

I've been continuing the work on trying to get VMs running properly in general, and so far as I'm aware Proxmox uses the same magic under the hood that LXD does so if we make it work on one it's very likely to work on the other.

At the moment I've got VM creation working properly but am stuck at the "booting" phase of things as no OS I've tried to date seems to have made it to even initializing the display. On a positive note though, I have been able to fiddle with libtpm and swtpm to overcome the secureboot issue, so there's at least that I suppose...

I'll keep the thread updated as I find / work out more.

[Sysadminfromhell]

@Sysadminfromhell I've been fiddling with my three-node cluster since this thread was opened and have not once yet been able to so much as get PiMox to install properly (again). Something goes wrong literally every single time, most often the ZFS module.

It became so frustrating that I eventually just wiped all three nodes and burned fresh Ubuntu Server 21.10 arm64 images onto the drives and manually set up a three-node LXD cluster. It's way less convenient than Proxmox, but so far I can confirm that it at least seems stable.

I've been continuing the work on trying to get VMs running properly in general, and so far as I'm aware Proxmox uses the same magic under the hood that LXD does so if we make it work on one it's very likely to work on the other.

At the moment I've got VM creation working properly but am stuck at the "booting" phase of things as no OS I've tried to date seems to have made it to even initializing the display. On a positive note though, I have been able to fiddle with libtpm and swtpm to overcome the secureboot issue, so there's at least that I suppose...

I'll keep the thread updated as I find / work out more.

If you have time we can meet in Discord and can work this through together if you have a Pi to spear.

just message me here directly.

kind regards,

EDIT:

@the-wondersmith, just messaging me on discord: FourK#5783

[TuxfeatMac]

@the-wondersmith you could try the DietPi installer. https://github.com/TuxfeatMac/pimox7-installer/tree/main/dietpi There the installation runs through without any issues. After rebbot there is nothing to do, just visit the web interface.

[the-wondersmith]

@TuxfeatMac As soon as I'm able I'll try the dietpi install. I've hooked up with @Sysadminfromhell via Discord. I'll update this thread with any progress we make.

[Sysadminfromhell]

Can we get Windows somehow to work? If I try and run through the Wizard it cant start the VM:

kvm: -no-hpet: Option not supported for this target TASK ERROR: start failed: QEMU exited with code 1

VM-Config:

EDIT:

Nevermind, could fix it with adding arch: x86_64 to the config 100.

EDIT2: Installation works but cant boot into it. Removed the VM for now.

I got a x86_64 Maschine working (Windows Server 2k16 Nano). Its slow but it gets the job done, what i think is really suprising is that i gave it 4 vCPU and its always just using about 25% (1 Core). I will install all the drivers and see if this will fix it.

EDIT: I just abort the whole thing, it's too slow and the whole system seems unstable / freezes a lot. Even after 4 hours of installing the drivers the system was just using 30% of 4 cores. I guess that shows that you shouldn't emulate an x86_64 on a ARM64. It doesn't go well. In this case we can't use this architecture.

[the-wondersmith]

Update:

I haven't been able to test the new pimox-specific kernel or installation from dietpi. I have (however) been able to get Windows 11 to install properly and run at a usable speed under QEMU. The full procedure was a MA-HOOSIVE pain in the ass, but the VM now starts and runs stably.

I've packed up the image and sent it to @Sysadminfromhell for testing, to ensure that it working properly wasn't just some fluke or stroke of luck on my part. I'm currently waiting for parts so I can create a pimox boot disk without losing my current (nicely functional) ubuntu-based system. Whenever said parts arrive (and assuming the image runs properly for him whenever he gets around to it) I'll see if I can recreate my success under pimox and type up a proper guide on how I did.

I did figure a few things out though that I'm at a loss for how to begin investigating, most prominent being the fact that I encountered a similar lagging issue to the one reported by @Sysadminfromhell but noticed that spamming info blockstats in the QEMU console seemed to cause them the completely disappear - almost as though there's some kind of weirdness around I/O priority or scheduling that causes the VM to get knocked down to the bottom of the list normally but popped back to the top whenever its stats are requested.

Any ideas?

[auvio323]

Is it possible to run the ARM version of Windows 11? I get the following error when trying to do so:

kvm: -no-hpet: Option not supported for this target TASK ERROR: start failed: QEMU exited with code 1

It seems that this error is an AMD processor option on Windows, which could explain why it is doing that. I'm just not sure how to remove it. When I change the OS to anything other to Windows, the NoVNC opens, but it freezes while booting.

Trying to enable TPM gives me the following error: TASK ERROR: start failed: command 'swtpm_setup --tpmstate file:///var/lib/vz/images/101/vm-101-disk-3.raw --createek --create-ek-cert --create-platform-cert --lock-nvram --config /etc/swtpm_setup.conf --runas 0 --not-overwrite --tpm2 --ecc' failed: open3: exec of swtpm_setup --tpmstate file:///var/lib/vz/images/101/vm-101-disk-3.raw --createek --create-ek-cert --create-platform-cert --lock-nvram --config /etc/swtpm_setup.conf --runas 0 --not-overwrite --tpm2 --ecc failed: No such file or directory at /usr/share/perl5/PVE/Tools.pm line 455.

[dbzoo]

/usr/share/perl5/PVE/QemuServer.pm

    if ($winversion >= 6) {
        push @$globalFlags, 'kvm-pit.lost_tick_policy=discard';
        push @$cmd, '-no-hpet';
    }

-no-hpet is added as the ostype has a version >= 6 the version becomes the $winversion. For example you might have configured this in /etc/pve/qemu-server/nodeid.conf file ostype: win11