The -r img option fails on arm64

[ikitayama]

I can not make sense of this:

dracut: Executing: /usr/bin/dracut --uefi --kver 5.15.0-25-generic --kernel-cmdline "rhgb selinux=0 audit=0 rw" --force /efi/EFI/Linux/linux-5.15.0-25-generic.efi
dracut: Architecture 'aarch64' not supported to create a UEFI executable
‣ Error: Workspace command /etc/kernel/install.d/50-mkosi-dracut-unified-kernel.install add 5.15.0-25-generic /efi/c2627301643043e6a22c9abfbbb80b7a/5.15.0-25-generic '' returned non-zero exit code 1.
‣  (Unmounting image)
‣  (Detaching /dev/loop32)

This is seen when I run run_qemu.sh with the options:

qemu=$HOME/projects/qemu/build/qemu-system-aarch64 run_qemu.sh -r img --no-cxl --no-kvm  --no-hmat --pmems 0 -p 1S

My goal is to test the cxl_test module, but I am stuck on this.

[ikitayama]

I'm on Ubuntu 22.04 LTS and uses mkosi included of the release, while dracut is built from the source.

[stellarhopper]

@ikitayama To rule out any oddness with the self built dracut, can you try with the system provided version of it too?

[ikitayama]

@ikitayama To rule out any oddness with the self built dracut, can you try with the system provided version of it too?

Sure, I did sudo make install from the dracut git cloned directory and did reinstall the dracut package.

dracut: Executing: /usr/bin/dracut --uefi --kver 6.14.0-rc1-gaae0594a7053 --kernel-cmdline "rhgb selinux=0 audit=0 rw" --force /efi/EFI/Linux/linux-6.14.0-rc1-gaae0594a7053.efi
dracut: Architecture 'aarch64' not supported to create a UEFI executable
‣ Error: Workspace command /etc/kernel/install.d/50-mkosi-dracut-unified-kernel.install add 6.14.0-rc1-gaae0594a7053 /efi/8ddab8ce22d24140b365bc37f0e9b2f3/6.14.0-rc1-gaae0594a7053 '' returned non-zero exit code 1.

[ikitayama]

This time I started run_qemu.sh with the -r wipe option in the Linux kernel source code (builddir) directory.

[ikitayama]

Here are the changes localy I made for the script to run on the arm64 Linux port.

diff --git a/run_qemu.sh b/run_qemu.sh
index 60642ba..2d89aa3 100755
--- a/run_qemu.sh
+++ b/run_qemu.sh
@@ -445,7 +445,8 @@ make_install_kernel()
 		exit 1
 	}
 
-	cat arch/x86_64/boot/bzImage > "$inst_path"/vmlinuz-"$kver"
+	#cat arch/x86_64/boot/bzImage > "$inst_path"/vmlinuz-"$kver"
+	cat arch/arm64/boot/Image > "$inst_path"/vmlinuz-"$kver"
 	cp System.map "$inst_path"/System.map-"$kver"
 	ln -fs vmlinuz-"$kver" "$inst_path"/vmlinuz
 	ln -fs System.map-"$kver" "$inst_path"/System.map
@@ -1376,6 +1377,7 @@ setup_nvme()
 		qcmd+=("-device" "nvme,drive=nvme$i,serial=deadbeaf$i${extra_args}")
 		qcmd+=("-drive" "file=$nvme_img,if=none,id=nvme$i")
 	done
+	qcmd+=("-device acpi-ged,event=memory-hotplug")
 }
 
 setup_cxl()
@@ -1509,7 +1511,8 @@ prepare_qcmd()
 	if [[ $_arg_kvm = "off" ]]; then
 		accel="tcg" # the default
 	fi
-	machine_args=("q35" "accel=$accel")
+	#machine_args=("q35" "accel=$accel")
+	machine_args=("virt" "accel=$accel")
 	if [[ "$num_pmems" -gt 0 ]]; then
 		machine_args+=("nvdimm=on")
 	fi
@@ -1530,7 +1533,7 @@ prepare_qcmd()
 		get_ovmf_binaries
 		qcmd+=("-drive" "if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly=on")
 		qcmd+=("-drive" "if=pflash,format=raw,unit=1,file=OVMF_VARS.fd")
-		qcmd+=("-debugcon" "file:uefi_debug.log" "-global" "isa-debugcon.iobase=0x402")
+		#qcmd+=("-debugcon" "file:uefi_debug.log" "-global" "isa-debugcon.iobase=0x402")
 	fi
 	qcmd+=("-drive" "file=$_arg_rootfs,format=raw,media=disk")
 	if [ $_arg_direct_kernel = "on" ] && [ -n "$vmlinuz" ] && [ -n "$initrd" ]; then
@@ -1579,7 +1582,8 @@ prepare_qcmd()
 	for (( i = 0; i < num_nodes; i++ )); do
 		[[ $_arg_hmat == "on" ]] && hmat_append="initiator=$i"
 		qcmd+=("-object" "memory-backend-ram,id=mem${i},size=${_arg_mem_size}M")
-		qcmd+=("-numa" "node,nodeid=${i},memdev=mem${i},$hmat_append")
+		#qcmd+=("-numa" "node,nodeid=${i},memdev=mem${i},$hmat_append")
+		qcmd+=("-numa" "node,nodeid=${i},memdev=mem${i}")
 		qcmd+=("-numa" "cpu,node-id=${i},socket-id=${i}")
 	done

[ikitayama]

@stellarhopper et al., do you guys work in a Fedora and its variants environment, by the way? I see the packaging of mkosi is slightly different between Fedora and Ubuntu.

[marc-hb]

Try WithUnifiedKernelImages=no, see https://github.com/systemd/mkosi/blob/v12/man/mkosi.1.

I'm on Ubuntu 22.04 LTS and uses mkosi included of the release,

Both are crazy old, you should upgrade. See run_qemu/README.md to upgrade mkosi only.

do you guys work in a Fedora and its variants environment, by the way?

I've been using Ubuntu, Fedora and Arch and all of them more or less work but using a 3 years old LTS is not great. mkosi developers try to maintain all distributions but:

• they (mkosi) don't care one bit about old bugs and old releases
• I think they use Fedora primarily?

[ikitayama]

@marc-hb thanks, I had to be on the ancient 22.04 LTS for other project(s), but I'll see if I can observe the same situation with the latest Ubuntu release be it LTS or not.

[ikitayama]

This what I see on Ubuntu 24.04

[...]
Note, selecting 'libtss2-esys-3.0.2-0' for regex '^libtss2-esys-[0-9\.]+-0$'
Note, selecting 'libtss2-esys-3.0.2-0t64' instead of 'libtss2-esys-3.0.2-0'
Note, selecting 'libtss2-rc0t64' instead of 'libtss2-rc0'
Note, selecting 'libtss2-tcti-device0t64' instead of 'libtss2-tcti-device0'
Package libtss2-mu0 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  libtss2-mu-4.0.1-0t64

E: Package 'libtss2-mu0' has no installation candidate
‣ "env APT_CONFIG=/home/realm/.cache/mkosi-workspacecr88sa2w/apt.conf DEBIAN_FRONTEND=noninteractive DEBCONF_INTERACTIVE_SEEN=true INITRD=No apt-get -o APT::Architecture=arm64 -o APT::Architectures=arm64 -o APT::Install-Recommends=false -o APT::Immediate-Configure=off -o APT::Get::Assume-Yes=true -o APT::Get::AutomaticRemove=true -o APT::Get::Allow-Change-Held-Packages=true -o APT::Get::Allow-Remove-Essential=true -o APT::Sandbox::User=root -o Dir::Cache=/var/cache/apt -o Dir::State=/var/lib/apt -o Dir::State::Status=/home/realm/.cache/mkosi-workspacecr88sa2w/root/var/lib/dpkg/status -o Dir::Log=/home/realm/.cache/mkosi-workspacecr88sa2w -o Dir::Bin::DPkg=/usr/bin/dpkg -o Debug::NoLocking=true -o DPkg::Options::=--root=/home/realm/.cache/mkosi-workspacecr88sa2w/root -o DPkg::Options::=--force-unsafe-io -o DPkg::Options::=--force-architecture -o DPkg::Options::=--force-depends -o DPkg::Options::=--no-debsig -o DPkg::Use-Pty=false -o DPkg::Install::Recursive::Minimum=1000 -o pkgCacheGen::ForceEssential=, -o 'DPkg::Options::=--path-exclude=/usr/share/doc/*' -o 'DPkg::Options::=--path-include=/usr/share/doc/*/copyright' -o 'DPkg::Options::=--path-exclude=/usr/share/man/*' -o 'DPkg::Options::=--path-exclude=/usr/share/groff/*' -o 'DPkg::Options::=--path-exclude=/usr/share/info/*' install '^libtss2-esys-[0-9\.]+-0$' bash dmsetup e2fsprogs kmod less libfido2-1 libtss2-mu0 libtss2-rc0 libtss2-tcti-device0 lvm2 p11-kit systemd udev util-linux" returned non-zero exit code 100.
tput: unknown terminal "xterm-ghostty"
tput: unknown terminal "xterm-ghostty"

[ikitayama]

/var/tmp/repart-EIqbSh successfully formatted as vfat (label "ESP", uuid f17feb2f)
Successfully formatted future partition 0.
Adding new partition 0 to partition table.
Writing new partition table.
All done.
‣  /home/realm/projects/cxl/qbuild/root.img.raw size is 2.8G, consumes 2.5G.
‣  Fixing ownership of package manager cache directory
Disk /dev/loop0: 2.75 GiB, 2954559488 bytes, 5770624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C08CA442-BD4C-4942-AAF1-39E839FC598D

Device         Start     End Sectors  Size Type
/dev/loop0p1    2048 1050623 1048576  512M EFI System
/dev/loop0p2 1050624 5770583 4719960  2.3G Linux root (ARM-64)
default run-qemu-kernel-6.14.0-rc1-gaae0594a7053.conf
cp: cannot stat '/usr/share/OVMF//Shell.efi': No such file or directory
cp: cannot stat '/usr/share/edk2-shell/x64/Shell_Full.efi': No such file or directory
OVMF binaries not found, please install '[edk2-]ovmf' or similar, 'edk2-shell', ...

Don't understand why this mkosi, if I am not mistaken, wants to stat X86 firmware. I'll look for the mkosi conf files.

[stellarhopper]

Here are the changes localy I made for the script to run on the arm64 Linux port.

diff --git a/run_qemu.sh b/run_qemu.sh
index 60642ba..2d89aa3 100755
--- a/run_qemu.sh
+++ b/run_qemu.sh
@@ -445,7 +445,8 @@ make_install_kernel()

<snip>

@ikitayama These look useful generally - once you have the mkosi stuff sorted out, it could be useful to include these changes under an option. Maybe --arch={x86_64,arm64}, defaulting to x86_64 ?

[marc-hb]

E: Package 'libtss2-mu0' has no installation candidate

See hack in commit 8d3d469c9f5c for this error. Or, you can upgrade mkosi instead, check run_qemu/README.md. I've been using export mkosi_bin=/your/mkosi.git/bin a lot and it's been working fine.

cp: cannot stat '/usr/share/OVMF//Shell.efi': No such file or directory

The OVMF part of run_qemu.sh is a big mess[*] and it's hardcoded to X86 right now. You can either de-hardcode (patches welcome!), or you can try to avoid OVMF completely. Try either this:

@@ -1539,7 +1540,7 @@ prepare_qcmd()
 	if [[ $_arg_log ]]; then
 		qcmd+=("-serial" "file:$_arg_log")
 	fi
-	if [[ $_arg_legacy_bios == "off" ]] ; then
+	if [[ $_arg_legacy_bios == "off" ]] || [[ $_arg_direct_kernel = "on" ]] ; then
 		get_ovmf_binaries
 		qcmd+=("-drive" "if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly=on")
 		qcmd+=("-drive" "if=pflash,format=raw,unit=1,file=OVMF_VARS.fd")

or --legacy-bios=on.

[*] to be fair, every distro packages OVMF differently. That does not help.

[ikitayama]

I confirm that on arm64 the "-r img" option (and none) works.

• Upgrading to the latest Ubuntu release 24.10.
• Use the mkosi tool on the cutting edge.
• Run the run_qemu.sh script on a VM that have enough disk size to cover the default 10GB.
• Minor and easy to spot run_qemu.sh script editing.

[ikitayama]

BTW, I still need solve this:

qemu-system-aarch64: -device nvdimm,memdev=nvmem0,id=nv0,label-size=2M,node=4: memory hotplug is not enabled: missing acpi-ged device

The binary 24.10 provides does not have it. (only acpi-erst)

[ikitayama]

E: Package 'libtss2-mu0' has no installation candidate

See hack in commit 8d3d469 for this error. Or, you can upgrade mkosi instead, check run_qemu/README.md. I've been using export mkosi_bin=/your/mkosi.git/bin a lot and it's been working fine.

cp: cannot stat '/usr/share/OVMF//Shell.efi': No such file or directory

The OVMF part of run_qemu.sh is a big mess[*] and it's hardcoded to X86 right now. You can either de-hardcode (patches welcome!), or you can try to avoid OVMF completely. Try either this:

@@ -1539,7 +1540,7 @@ prepare_qcmd() if [[ $_arg_log ]]; then qcmd+=("-serial" "file:$_arg_log") fi

• if [[ $_arg_legacy_bios == "off" ]] ; then

• if [[ $_arg_legacy_bios == "off" ]] || [[ $_arg_direct_kernel = "on" ]] ; then get_ovmf_binaries qcmd+=("-drive" "if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly=on") qcmd+=("-drive" "if=pflash,format=raw,unit=1,file=OVMF_VARS.fd") or --legacy-bios=on.

[*] to be fair, every distro packages OVMF differently. That does not help.

On Ubuntu 24.10, they're AAVMF_CODE.fd and AAVMF_VARS.fd and located in the /usr/share/AAVMF directory.

+get_aavmf_binaries()
+{
+       if ! [ -e "AAVMF_CODE.fd" ] && ! [ -e "AAVMF_VARS.fd" ]; then
+                if [ ! -f "$aavmf_path/AAMVF_CODE.fd" ]; then
+                        echo "AAVMF binaries not found, please install '[edk2-]ovmf' or similar, '
edk2-shell', ..."
+                        exit 1
+                fi
+                cp "$aavmf_path/AAVMF_CODE.fd" .
+                cp "$aavmf_path/AAVMF_VARS.fd" .
        fi
 }

Adding something like above sounds okay to you? I still don't make sense of how ubuntu_vars.sh gets used together with the run_qemu.sh though.

[ikitayama]

@stellarhopper @marc-hb I made a PR, please have a look at it. Thanks!

EDIT: #140 and more

[marc-hb]

@@ -1539,7 +1540,7 @@ prepare_qcmd()
 	if [[ $_arg_log ]]; then
 		qcmd+=("-serial" "file:$_arg_log")
 	fi
-	if [[ $_arg_legacy_bios == "off" ]] ; then
+	if [[ $_arg_legacy_bios == "off" ]] || [[ $_arg_direct_kernel = "on" ]] ; then
 		get_ovmf_binaries
 		qcmd+=("-drive" "if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly=on")
 		qcmd+=("-drive" "if=pflash,format=raw,unit=1,file=OVMF_VARS.fd")

Sorry that was wrong. @ikitayama could you try #150 instead? It would be great if we can postpone the entire OVMF/AAVMF initially and get something working and merged faster.

EDIT: forgot to say, --direct-kernel is the default.

[marc-hb]

@ikitayama I feel we've gone a long way since you filed this, please close if you can't find anything still relevant.

[ikitayama]

@marc-hb thanks to your major overhaul of the script, I think this won't come up for arm64. Closing.