lima-vm: vz error with the pkgx version, not with the brew one

[tannevaled]

Nested virtualization under M3+ masOS 15+

Template file : nested.yaml

minimumLimaVersion: "1.0.0"
images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20241004/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:fad101d50b06b26590cf30542349f9e9d3041ad7929e3bc3531c81ec27f2c788"
- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20241004/ubuntu-24.04-server-cloudimg-arm64.img"
  arch: "aarch64"
  digest: "sha256:e380b683b0c497d2a87af8a5dbe94c42eb54548fa976167f307ed8cf3944ec57"
# Fallback to the latest release image.
# Hint: run `limactl prune` to invalidate the cache
- location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img"
  arch: "aarch64"

mounts:
- location: "~"
- location: "/tmp/lima"
  writable: true

vmType: vz
nestedVirtualization: true

VM creation

phymath@mba-10838921 Documents % pkgx limactl create --name nested --tty=false ./nested.yaml
INFO[0000] Terminal is not available, proceeding without opening an editor 
WARN[0000] vmType vz: ignoring [User]                   
INFO[0000] Attempting to download the image              arch=aarch64 digest="sha256:e380b683b0c497d2a87af8a5dbe94c42eb54548fa976167f307ed8cf3944ec57" location="https://cloud-images.ubuntu.com/releases/24.04/release-20241004/ubuntu-24.04-server-cloudimg-arm64.img"
INFO[0000] Using cache "/Users/phymath/Library/Caches/lima/download/by-url-sha256/1f3456e7d7c2bc0a8f8993d1308a3a4c124d703f56bf4cf6dfca50eb5f11f1c3/data" 
INFO[0000] Converting "/Users/phymath/.lima/nested/basedisk" (qcow2) to a raw disk "/Users/phymath/.lima/nested/diffdisk" 
3.50 GiB / 3.50 GiB [---------------------------------------] 100.00% 1.62 GiB/s
INFO[0002] Expanding to 100GiB                          
INFO[0002] Attempting to download the nerdctl archive    arch=aarch64 digest="sha256:fe085381a09aa240ae5d1e0bbef1beccfb7c1d6dbb98bdc55bd416581d46ebc8" location="https://github.com/containerd/nerdctl/releases/download/v2.0.0/nerdctl-full-2.0.0-linux-arm64.tar.gz"
INFO[0002] Using cache "/Users/phymath/Library/Caches/lima/download/by-url-sha256/1699e54a52757df863155fca76f8a77b50f05d993edca23421798af6635156f0/data" 
INFO[0002] Run `limactl start nested` to start the instance.

VM start (pkgx version)

phymath@mba-10838921 Documents % pkgx limactl start nested                             
INFO[0000] Using the existing instance "nested"         
INFO[0000] Starting the instance "nested" with VM driver "vz" 
WARN[0000] vmType vz: ignoring [User]                   
INFO[0000] [hostagent] hostagent socket created at /Users/phymath/.lima/nested/ha.sock 
INFO[0000] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/phymath/.lima/nested/serial*.log") 
FATA[0001] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/phymath/.lima/nested/ha.stderr.log") 

phymath@mba-10838921 Documents % grep features /Users/phymath/.lima/nested/ha.stderr.log
{"level":"debug","msg":"Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.","time":"2024-11-07T15:33:54+01:00"}

VM start (brew version)

phymath@mba-10838921 Documents % limactl start nested    
INFO[0000] Using the existing instance "nested"         
INFO[0000] Starting the instance "nested" with VM driver "vz" 
WARN[0000] vmType vz: ignoring [User]                   
INFO[0000] [hostagent] hostagent socket created at /Users/phymath/.lima/nested/ha.sock 
INFO[0000] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/phymath/.lima/nested/serial*.log") 
INFO[0001] SSH Local Port: 57631                        
INFO[0001] [hostagent] [VZ] - vm state change: running  
INFO[0001] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
INFO[0011] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
INFO[0011] [hostagent] The essential requirement 1 of 2 is satisfied 
INFO[0011] [hostagent] Waiting for the essential requirement 2 of 2: "user session is ready for ssh" 
INFO[0011] [hostagent] The essential requirement 2 of 2 is satisfied 
INFO[0011] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available" 
INFO[0011] [hostagent] Guest agent is running           
INFO[0011] [hostagent] Not forwarding UDP 127.0.0.54:53 
INFO[0011] [hostagent] Not forwarding UDP 127.0.0.53:53 
INFO[0011] [hostagent] Not forwarding UDP 192.168.5.15:68 
INFO[0011] [hostagent] Not forwarding TCP 127.0.0.53:53 
INFO[0011] [hostagent] Not forwarding TCP 127.0.0.54:53 
INFO[0011] [hostagent] Not forwarding TCP [::]:22       
INFO[0011] [hostagent] The optional requirement 1 of 2 is satisfied 
INFO[0011] [hostagent] Waiting for the optional requirement 2 of 2: "containerd binaries to be installed" 
INFO[0023] [hostagent] The optional requirement 2 of 2 is satisfied 
INFO[0023] [hostagent] Waiting for the guest agent to be running 
INFO[0023] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished" 
INFO[0026] [hostagent] Forwarding TCP from 127.0.0.1:44307 to 127.0.0.1:44307 
INFO[0035] [hostagent] The final requirement 1 of 1 is satisfied 
INFO[0035] READY. Run `limactl shell nested` to open the shell. 

phymath@mba-10838921 Documents % limactl shell nested
phymath@lima-nested:/Users/phymath/Documents$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

phymath@mba-10838921 Documents % pkgx limactl --version
limactl version 1.0.0
phymath@mba-10838921 Documents % limactl --version
limactl version 1.0.0
phymath@mba-10838921 Documents % which limactl
/opt/homebrew/bin/limactl

[jhheider]

Does ha.stderr.log show anything interesting? It looks like it's trying to start ssh and failing. Maybe we need openssh.org as a companion? I don't see any particular major differences with https://formulae.brew.sh/formula/lima, unless we're not getting the templates installed, maybe?

[jhheider]

I get {"level":"fatal","msg":"nested virtualization is not supported on this device","time":"2024-11-07T14:53:01-05:00"}, so that suggests we need a build flag or library. Might even be a qemu issue.

[jhheider]

actually:

{"level":"debug","msg":"ResolveVMType: resolved VMType \"vz\" (explicitly specified in []*LimaYAML{o,y,d}[1])","time":"2024-11-07T14:56:59-05:00"}
{"level":"debug","msg":"Creating iso file /Users/jacob/.lima/nested/cidata.iso","time":"2024-11-07T14:56:59-05:00"}
{"level":"debug","msg":"Using /var/folders/8x/k382fgcs59vfffl1dgq015gh0000gn/T/diskfs_iso115028808 as workspace","time":"2024-11-07T14:56:59-05:00"}
{"level":"debug","msg":"Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"OpenSSH version 9.8.1 detected","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing [email protected] and [email protected]","time":"2024-11-07T14:57:00-05:00"}
{"level":"info","msg":"hostagent socket created at /Users/jacob/.lima/nested/ha.sock","time":"2024-11-07T14:57:00-05:00"}
{"level":"info","msg":"Starting VZ (hint: to watch the boot progress, see \"/Users/jacob/.lima/nested/serial*.log\")","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Start udp DNS listening on: 127.0.0.1:60329","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Using search domains: [jacobsdomain.arpa]","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Start tcp DNS listening on: 127.0.0.1:64960","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Kernel file \"/Users/jacob/.lima/nested/kernel\" not found","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Using EFI Boot Loader","time":"2024-11-07T14:57:00-05:00"}
{"level":"fatal","msg":"nested virtualization is not supported on this device","time":"2024-11-07T14:57:00-05:00"}

it finds AES, but doesn't find the kernel. that seems like it's potentially a problem.

[jhheider]

i note that nestedVirtualization is off in pkgx limactl info. Can you diff the info output of the two? it might be that we need some build flags.

[jhheider]

our qemu is missing --enable-fdt=system. checking to see if that might be contributory.

[jhheider]

interestingly, it works with pkgx limactl~0.23.2 start nested, though it complains about the nestedVirtualization key. so, it seems like it's either something to do with our build for v1, or something that v0 isn't checking in qemu (but i don't see what it might be). more exploration definitely needed.

[tannevaled]

on osx it does not use qemu but vz. i will have acces to the M3 hardware next week to test what you asked.

[jhheider]

cool. yeah, always good to have users involved in testing. i reviewed both their release process and the homebrew build and didn't see differences of note, though there's clearly something.

[tannevaled]

i note that nestedVirtualization is off in pkgx limactl info. Can you diff the info output of the two? it might be that we need some build flags.

nestedvirtualization is off even when using the brew limactl binary. when diffing the info output only the templates location differ.

[tannevaled]

on what kind of hardware is it built? M3- or M3+ ?

[tannevaled]

cf https://github.com/lima-vm/lima/blob/master/.github/workflows/release.yml and https://github.com/lima-vm/lima/issues/2767 the compiling environment is important to enable the new features

[jhheider]

M1. We should be able to control that with build flags, hopefully.

[jhheider]

So, this appears to be the relevant code: https://github.com/lima-vm/lima/blob/5ac7de0bf9e45e403f1af08d1f2f998bb8d04d58/Makefile#L24-L31

Both build machines are using the 14.5 SDK, so it shouldn't be disabling vz.

[jhheider]

I find no difference in behavior building with sdk 14.5 or 15.1 locally (though, I also have the brew binary fail to nest on my M2). the entitlements look correct too:

$ codesign -d --entitlements - ~/.pkgx/lima-vm.io/v1.0.1/bin/limactl
Executable=/Users/jacob/.pkgx/lima-vm.io/v1.0.1/bin/limactl
[Dict]
	[Key] com.apple.security.network.client
	[Value]
		[Bool] true
	[Key] com.apple.security.network.server
	[Value]
		[Bool] true
	[Key] com.apple.security.virtualization
	[Value]
		[Bool] true

[tannevaled]

the nested only works on M3(+)