Failes to find binfmt_misc and build under Ubuntu 21.04

[dodbrian]

I'm trying to follow instructions in the README.md and running example files for Raspberry Pi 4. Every time the build fails with an error: ==> arm: Failed to find binfmt_misc for /usr/bin/qemu-aarch64-static under /proc/sys/fs/binfmt_misc.

After several attempts I decided to run the vagrant version and everything worked just fine. The Vagrantfile uses Ubuntu Focal as a base image:

config.vm.box = "ubuntu/focal64"

I changed it to Ubuntu Hirsute:

config.vm.box = "ubuntu/hirsute64"

and now the bug is reproducible in a VM:

vagrant@ubuntu-hirsute:~/packer-builder-arm$ sudo packer build boards/raspberry-pi-4/ubuntu_server_20.04_arm64.json 
arm: output will be in this color.

==> arm: Retrieving rootfs_archive
==> arm: Trying http://cdimage.ubuntu.com/releases/20.04.2/release/ubuntu-20.04.2-preinstalled-server-arm64+raspi.img.xz?archive=false
==> arm: Trying http://cdimage.ubuntu.com/releases/20.04.2/release/ubuntu-20.04.2-preinstalled-server-arm64+raspi.img.xz?archive=false&checksum=file%3Ahttp%3A%2F%2Fcdimage.ubuntu.com%2Freleases%2F20.04.2%2Frelease%2FSHA256SUMS
    arm: ubuntu-20.04.2-preinstalled-server-arm64+raspi.img.xz 674.63 MiB / 674.63 MiB [====================================================] 100.00% 1m0s
==> arm: http://cdimage.ubuntu.com/releases/20.04.2/release/ubuntu-20.04.2-preinstalled-server-arm64+raspi.img.xz?archive=false&checksum=file%3Ahttp%3A%2F%2Fcdimage.ubuntu.com%2Freleases%2F20.04.2%2Frelease%2FSHA256SUMS => /home/vagrant/packer-builder-arm/packer_cache/d6cf9f19f60b9a1d4a73e31d910f4d906ff2bfb5.xz
    arm: unpacking /home/vagrant/packer-builder-arm/packer_cache/d6cf9f19f60b9a1d4a73e31d910f4d906ff2bfb5.xz to ubuntu-20.04.img
    arm: unpacking with custom comand: [xz --decompress /tmp/image483344427/d6cf9f19f60b9a1d4a73e31d910f4d906ff2bfb5.xz]
    arm: searching for empty loop device (to map ubuntu-20.04.img)
    arm: mapping image ubuntu-20.04.img to /dev/loop3
    arm: mounting /dev/loop3p2 to /tmp/161902990
    arm: mounting /dev/loop3p1 to /tmp/161902990/boot/firmware
    arm: running extra setup
    arm: mounting /dev with: [mount --bind /dev /tmp/161902990/dev]
    arm: mounting /devpts with: [mount -t devpts /devpts /tmp/161902990/dev/pts]
    arm: mounting proc with: [mount -t proc proc /tmp/161902990/proc]
    arm: mounting binfmt_misc with: [mount -t binfmt_misc binfmt_misc /tmp/161902990/proc/sys/fs/binfmt_misc]
    arm: mounting sysfs with: [mount -t sysfs sysfs /tmp/161902990/sys]
==> arm: Failed to find binfmt_misc for /usr/bin/qemu-aarch64-static under /proc/sys/fs/binfmt_misc
==> arm: remove /tmp/161902990/usr/bin/qemu-aarch64-static: no such file or directory
    arm: optional (please ignore) `fuser -k` failed with exit status 1:
Build 'arm' errored after 2 minutes 36 seconds: build was halted

==> Wait completed after 2 minutes 36 seconds

==> Some builds didn't complete successfully and had errors:
--> arm: build was halted

==> Builds finished but no artifacts were created.

[lcividin]

Hey for me the fix was I need to copy the conf files that are in one of these dirs into the other dir below (make the dirs if they're not there)

I just copied all the files but I think you really only need qemu-arm-static.conf for raspberry pi/arm

/lib/binfmt.d /usr/lib/binfmt.d/ /etc/binfmt.d/

then sudo systemctl restart systemd-binfmt.service and you should be golden.

https://manpages.ubuntu.com/manpages/cosmic/man5/binfmt.d.5.html

Got on the right track with this fellows explanation! (https://github.com/computermouth/qemu-static-conf ) Hope it helps!

edit: should also say I'm on popOS 21.04 which is based of Ubuntu

[goldyfruit]

Hey for me the fix was I need to copy the conf files that are in one of these dirs into the other dir below (make the dirs if they're not there)

I just copied all the files but I think you really only need qemu-arm-static.conf for raspberry pi/arm

/lib/binfmt.d /usr/lib/binfmt.d/ /etc/binfmt.d/

then sudo systemctl restart systemd-binfmt.service and you should be golden.

https://manpages.ubuntu.com/manpages/cosmic/man5/binfmt.d.5.html

Got on the right track with this fellows explanation! (https://github.com/computermouth/qemu-static-conf ) Hope it helps!

edit: should also say I'm on popOS 21.04 which is based of Ubuntu

Are you still able to make it works on PopOS 22.04?

[lcividin]

Hey for me the fix was I need to copy the conf files that are in one of these dirs into the other dir below (make the dirs if they're not there) I just copied all the files but I think you really only need qemu-arm-static.conf for raspberry pi/arm /lib/binfmt.d /usr/lib/binfmt.d/ /etc/binfmt.d/ then sudo systemctl restart systemd-binfmt.service and you should be golden. https://manpages.ubuntu.com/manpages/cosmic/man5/binfmt.d.5.html Got on the right track with this fellows explanation! (https://github.com/computermouth/qemu-static-conf ) Hope it helps! edit: should also say I'm on popOS 21.04 which is based of Ubuntu

Are you still able to make it works on PopOS 22.04?

Hey I did up grade to 22.04 but I didn't try it, I've since moved to fedora 36 and not set it back up yet. I don't see why it wouldn't work on 22.04 though Sorry I couldn't be of more assistance

[goldyfruit]

Hey for me the fix was I need to copy the conf files that are in one of these dirs into the other dir below (make the dirs if they're not there) I just copied all the files but I think you really only need qemu-arm-static.conf for raspberry pi/arm /lib/binfmt.d /usr/lib/binfmt.d/ /etc/binfmt.d/ then sudo systemctl restart systemd-binfmt.service and you should be golden. https://manpages.ubuntu.com/manpages/cosmic/man5/binfmt.d.5.html Got on the right track with this fellows explanation! (https://github.com/computermouth/qemu-static-conf ) Hope it helps! edit: should also say I'm on popOS 21.04 which is based of Ubuntu

Are you still able to make it works on PopOS 22.04?

Hey I did up grade to 22.04 but I didn't try it, I've since moved to fedora 36 and not set it back up yet. I don't see why it wouldn't work on 22.04 though Sorry I couldn't be of more assistance

Thanks, no worries. :+1:

[yagop]

Not sure if is related to this but since Ubuntu 22.04 I had to change the qemu_binary_source_path.

       "image_chroot_env": ["PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"],
-      "qemu_binary_source_path": "/usr/bin/qemu-aarch64-static",
-      "qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static"
+      "qemu_binary_source_path": "/usr/libexec/qemu-binfmt/aarch64-binfmt-P",
+      "qemu_binary_destination_path": "/usr/libexec/qemu-binfmt/aarch64-binfmt-P"

$ cat /usr/share/binfmts/qemu-aarch64
package qemu-user-static
interpreter /usr/libexec/qemu-binfmt/aarch64-binfmt-P
magic \x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00
offset 0
mask \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff
credentials yes
fix_binary yes
preserve yes

Looks like this was changed since qemu (1:5.2+dfsg-4) see the changelog.

This changes the enterpreter name for all linux-user registered
    binfmts, so it potentially can break stuff.  The actual binary
    being registered now is /usr/libexec/qemu-binfmt/foo-binfmt-P,
    which is a symlink to actual /usr/lib/qemu-foo[-static].

[tbjers]

Looks like this was changed since qemu (1:5.2+dfsg-4) see the changelog.

Amazing find! I was trying to figure out what I was doing wrong. I tried in the Docker image, I tried in a VM, and I even reformatted one of my Intel NUCs to test this, and it kept failing until I updated the configuration files. This is definitely the solution.

[dbast]

Investigated this and /usr/libexec/qemu-binfmt/aarch64-binfmt-P is a symlink to /usr/bin/qemu-aarch64-static. So the solution is to evaluate symlinks when checking for file existence. This was fixed and is also tested with vagrant and Ubuntu 22.04 by this PR https://github.com/mkaczanowski/packer-builder-arm/pull/223. This means the board definitions even work on newer qemu versions without changes.