lima
lima copied to clipboard
lima-vm
Mount points do not resume after stop/start/reboot/etc of podman instance
Description
When using limactl with a podman instance and fuse sshfs for volume mounts, the podman instance will come up successfully just fine, but on subsequent attempts, it fails to mount the sshfs systems:
limactl stop podman
INFO[0000] Sending SIGINT to hostagent process 15141
INFO[0000] Waiting for the host agent and the qemu processes to shut down
INFO[0000] [hostagent] fusermount: mount failed: Operation not permitted
INFO[0000] [hostagent] fusermount: mount failed: Operation not permitted
INFO[0000] [hostagent] Received SIGINT, shutting down the host agent
INFO[0000] [hostagent] Shutting down the host agent
INFO[0000] [hostagent] Stopping forwarding "/run/user/502/podman/podman.sock" (guest) to "/Users/marcmercer/.lima/podman/sock/podman.sock" (host)
INFO[0000] [hostagent] Stopping forwarding "/run/lima-guestagent.sock" (guest) to "/Users/marcmercer/.lima/podman/ga.sock" (host)
INFO[0000] [hostagent] Unmounting "/Users/marcmercer"
INFO[0000] [hostagent] Unmounting "/tmp/lima"
INFO[0000] [hostagent] Shutting down QEMU with ACPI
INFO[0000] [hostagent] Sending QMP system_powerdown command
INFO[0001] [hostagent] QEMU has exited
limactl start podman
INFO[0000] Using the existing instance "podman"
INFO[0000] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/marcmercer/.lima/podman/serial.log")
INFO[0000] SSH Local Port: 53492
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0019] [hostagent] The essential requirement 1 of 5 is satisfied
INFO[0019] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0030] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0031] [hostagent] The essential requirement 2 of 5 is satisfied
INFO[0031] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0031] [hostagent] The essential requirement 3 of 5 is satisfied
INFO[0031] [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf to contain \"user_allow_other\""
INFO[0031] [hostagent] The essential requirement 4 of 5 is satisfied
INFO[0031] [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running"
INFO[0031] [hostagent] The essential requirement 5 of 5 is satisfied
INFO[0031] [hostagent] Mounting "/Users/marcmercer"
INFO[0031] [hostagent] fusermount: mount failed: Operation not permitted
WARN[0063] [hostagent] failed to confirm whether /Users/marcmercer [remote] is successfully mounted
INFO[0063] [hostagent] Mounting "/tmp/lima"
INFO[0063] [hostagent] fusermount: mount failed: Operation not permitted
WARN[0094] [hostagent] failed to confirm whether /tmp/lima [remote] is successfully mounted
INFO[0094] [hostagent] Waiting for the optional requirement 1 of 1: "user probe 1/1"
INFO[0094] [hostagent] Forwarding "/run/user/502/podman/podman.sock" (guest) to "/Users/marcmercer/.lima/podman/sock/podman.sock" (host)
INFO[0094] [hostagent] The optional requirement 1 of 1 is satisfied
INFO[0094] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/marcmercer/.lima/podman/ga.sock" (host)
INFO[0094] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0095] [hostagent] Not forwarding TCP 127.0.0.53:53
INFO[0095] [hostagent] Not forwarding TCP 0.0.0.0:22
INFO[0095] [hostagent] Not forwarding TCP [::]:22
INFO[0095] [hostagent] The final requirement 1 of 1 is satisfied
INFO[0095] READY. Run `limactl shell podman` to open the shell.
INFO[0095] Message from the instance "podman":
To run `podman` on the host (assumes podman-remote is installed):
$ export CONTAINER_HOST=unix:///Users/marcmercer/.lima/podman/sock/podman.sock
So far, the workaround is to delete and recreate the podman instance, forfeiting your currently built up infrastructure (images, containers, etc)
This is is SIMILAR to #584but not necessarily identical.
Looks like fusermount3 was upgraded by some buggy process, that failed to make it setuid.
That is why you are getting "Operation not permitted", since it is not running as user "root"
Healthy:
lrwxrwxrwx 1 root root 11 Jun 20 2021 /usr/bin/fusermount -> fusermount3
-rwsr-xr-x 1 root root 35200 Jun 20 2021 /usr/bin/fusermount3
Broken:
lrwxrwxrwx 1 root root 11 Jun 20 2021 /usr/bin/fusermount -> fusermount3
-rwxr-xr-x 1 root root 35200 Jun 20 2021 /usr/bin/fusermount3
There is also a "/etc/fuse.conf.dpkg-new" file, so maybe it was dpkg ?
The package "fuse3" is not part of the image, but installed on first boot.
Workaround (fixing the symptom):
sudo chmod u+s /bin/fusermount3
This is is SIMILAR to #584but not necessarily identical.
The other issue is related to docker starting before sshfs.
From /var/log/cloud-init-output.log:
Setting up fuse3 (3.10.3-2) ...
Configuration file '/etc/fuse.conf'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** fuse.conf (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package fuse3 (--configure):
end of file on stdin at conffile prompt
Setting up libavahi-glib1:amd64 (0.8-5ubuntu4) ...
dpkg: dependency problems prevent configuration of fuse-overlayfs:
fuse-overlayfs depends on fuse3; however:
Package fuse3 is not configured yet.
dpkg: error processing package fuse-overlayfs (--configure):
dependency problems - leaving unconfigured
The configure was supposed to set up the ownership too:
configure)
if [ -c /dev/cuse ] && ! chrooted
then
chmod 0600 /dev/cuse > /dev/null 2>&1
fi
if ! dpkg-statoverride --list /bin/fusermount3 > /dev/null 2>&1
then
chmod 4755 /bin/fusermount3
fi
modprobe fuse > /dev/null 2>&1 || true
if [ -x "`which update-initramfs 2>/dev/null`" ]
then
update-initramfs -u
fi
;;
There is also some conflict, between "sshfs" and "fuse-overlayfs":
Unpacking libfuse3-3:amd64 (3.10.3-2) ...
dpkg: fuse: dependency problems, but removing anyway as you requested:
sshfs depends on fuse.
ntfs-3g depends on fuse.
The workaround above does at least appear to work as a solution. Thanks @afbjorklund ;; creating a separate issue for the next observed issue.
I think the "default" image has the same issue now, after some recent upgrade.