multipass icon indicating copy to clipboard operation
multipass copied to clipboard

ssh connection failed: 'Socket create failed: Permission denied'

Open jclivi opened this issue 5 years ago • 19 comments

$ multipass mount ~/data/ ubuntu:/home/ubuntu/data mount failed: The following errors occurred:
error mounting "/home/ubuntu/data": Process returned exit code: 1: ssh connection failed: 'Socket create failed: Permission denied'

jclivi avatar Mar 11 '20 15:03 jclivi

Hi @jclivi, can you please post multipass info --all and also logs from journalctl -u snap.multipass*? Thanks!

Saviq avatar Mar 11 '20 15:03 Saviq

$ multipass info --all Name: ubuntu State: Running IPv4: 10.84.117.100 Release: Ubuntu 18.04.4 LTS Image hash: 3c3a67a14257 (Ubuntu 18.04 LTS) Load: 0.00 0.00 0.00 Disk usage: 1.1G out of 19.2G Memory usage: 79.3M out of 1.9G

jclivi avatar Mar 11 '20 17:03 jclivi

Also, it's not clear if you're using Multipass on Windows, OSX, or Linux. Could you please post the output of multipass version?

townsend2010 avatar Mar 11 '20 17:03 townsend2010

Linux run multipass

jclivi avatar Mar 11 '20 17:03 jclivi

deepin

jclivi avatar Mar 11 '20 17:03 jclivi

Also, if you're using Linux, could you please tell us the distro you are using? And if you're using OSX, which version of OSX?

townsend2010 avatar Mar 11 '20 17:03 townsend2010

-- Logs begin at Thu 2020-03-12 00:13:04 CST, end at Thu 2020-03-12 01:25:29 CST. -- jclivi multipassd[27032]: QMP: {"timestamp": {"seconds": 1583943853, "microseconds": 579021}, "event": "RTC_CHAN jclivi multipassd[27032]: mounting /data => /home/ubuntu/data in ubuntu jclivi multipassd[27032]: process program '/snap/multipass/1784/bin/sshfs_server' jclivi multipassd[27032]: process arguments '10.84.117.100, 22, ubuntu, /data, /home/ubunt jclivi multipassd[32430]: Applying AppArmor policy: multipass.ubuntu.1d81eb88.sshfs_server jclivi multipassd[27032]: ssh connection failed: 'Socket create failed: Permission denied' jclivi multipassd[27032]: Mount '/home/ubuntu/data' in instance "ubuntu" has stopped unexpectedly: Proc

jclivi avatar Mar 11 '20 17:03 jclivi

$ uname -a Linux jclivi 4.15.0-30deepin-generic #31 SMP Fri Nov 30 04:29:02 UTC 2018 x86_64 GNU/Linux

jclivi avatar Mar 11 '20 17:03 jclivi

lsb_release -a No LSB modules are available. Distributor ID: Deepin Description: Deepin 15.11 Release: 15.11 Codename: stable

jclivi avatar Mar 11 '20 17:03 jclivi

@jclivi when this happens, can you please have a look at the output of dmesg and see if there's any AppArmor DENIED messages that seem relevant?

Saviq avatar Mar 11 '20 17:03 Saviq

[ 3727.223417] audit: type=1400 audit(1583949073.731:572177): apparmor="ALLOWED" operation="recvmsg" profile="snap.microk8s.daemon-etcd" pid=6205 comm="etcd" laddr=::ffff:127.0.0.1 lport=12379 faddr=::ffff:127.0.0.1 fport=41674 family="inet6" sock_type="stream" protocol=6 requested_mask="receive" denied_mask="receive" [ 3727.223597] audit: type=1400 audit(1583949073.731:572178): apparmor="ALLOWED" operation="sendmsg" profile="snap.microk8s.daemon-etcd" pid=6205 comm="etcd" laddr=::ffff:127.0.0.1 lport=12379 faddr=::ffff:127.0.0.1 fport=41674 family="inet6" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send" [ 3727.223709] audit: type=1400 audit(1583949073.731:572179): apparmor="ALLOWED" operation="sendmsg" profile="snap.microk8s.daemon-etcd" pid=6205 comm="etcd" laddr=::ffff:127.0.0.1 lport=12379 faddr=::ffff:127.0.0.1 fport=41674 family="inet6" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send" [ 3727.223749] audit: type=1400 audit(1583949073.731:572180): apparmor="ALLOWED" operation="sendmsg" profile="snap.microk8s.daemon-etcd" pid=6205 comm="etcd" laddr=::ffff:127.0.0.1 lport=12379 faddr=::ffff:127.0.0.1 fport=41674 family="inet6" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send" [ 3727.223832] audit: type=1400 audit(1583949073.731:572181): apparmor="ALLOWED" operation="sendmsg" profile="snap.microk8s.daemon-etcd" pid=6205 comm="etcd" laddr=::ffff:127.0.0.1 lport=12379 faddr=::ffff:127.0.0.1 fport=41674 family="inet6" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send" jclivi@jclivi:~$ dmesg | grep "AppArmor" jclivi@jclivi:~$ dmesg | grep "DENIED"

jclivi avatar Mar 11 '20 17:03 jclivi

@Saviq Isn't ssh configured

jclivi avatar Mar 11 '20 17:03 jclivi

@jclivi this is quite weird, can you try:

$ sudo systemctl edit snap.multipass.multipassd.service
# add the following lines and save the file
# [Service]
# Environment=DISABLE_APPARMOR=1
$ snap restart multipass

And then try again?

The journal logs should say AppArmor disabled by environment variable.

Saviq avatar Mar 11 '20 18:03 Saviq

Ok, I installed Deepin in a VM and reproduced the behavior. These are the DENIED messages in dmesg:

[ 1275.966690] audit: type=1400 audit(1583953833.922:40235): apparmor="DENIED" operation="open" profile="multipass.test1.bf87f4b0.sshfs_server" name="/etc/ssh/ssh_config" pid=11695 comm="sshfs_server" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 1275.966694] audit: type=1400 audit(1583953833.922:40236): apparmor="DENIED" operation="create" profile="multipass.test1.bf87f4b0.sshfs_server" pid=11695 comm="sshfs_server" family="inet" sock_type="stream" protocol=6 requested_mask="create" denied_mask="create"
[ 1283.688488] audit: type=1400 audit(1583953841.642:40239): apparmor="DENIED" operation="open" profile="multipass.test1.bf87f4b0.sshfs_server" name="/etc/ssh/ssh_config" pid=11724 comm="sshfs_server" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 1283.688490] audit: type=1400 audit(1583953841.642:40240): apparmor="DENIED" operation="create" profile="multipass.test1.bf87f4b0.sshfs_server" pid=11724 comm="sshfs_server" family="inet" sock_type="stream" protocol=6 requested_mask="create" denied_mask="create"
[ 1294.335016] audit: type=1400 audit(1583953852.290:40243): apparmor="DENIED" operation="open" profile="multipass.test1.bf87f4b0.sshfs_server" name="/etc/ssh/ssh_config" pid=11753 comm="sshfs_server" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 1294.335132] audit: type=1400 audit(1583953852.290:40244): apparmor="DENIED" operation="create" profile="multipass.test1.bf87f4b0.sshfs_server" pid=11753 comm="sshfs_server" family="inet" sock_type="stream" protocol=6 requested_mask="create" denied_mask="create"

So it's definitely AppArmor denying this. Also, I tried the DISABLE_APPARMOR=1 and it works. I'll need to investigate what is needed in our AppArmor profile.

townsend2010 avatar Mar 11 '20 19:03 townsend2010

@townsend2010 $ sudo systemctl status snap.multipass.* ● snap.multipass.multipassd.service - Service for snap application multipass.multipassd Loaded: loaded (/etc/systemd/system/snap.multipass.multipassd.service; enabled; vendor prese Active: active (running) since Thu 2020-03-12 09:10:27 CST; 6min ago Main PID: 13857 (multipassd) Tasks: 9 (limit: 4915) Memory: 33.7M CPU: 1.590s CGroup: /system.slice/snap.multipass.multipassd.service └─13857 /snap/multipass/1784/bin/multipassd --verbosity debug --logger platform

3月 12 09:10:27 jclivi systemd[1]: Started Service for snap application multipass.multipassd. 3月 12 09:10:27 jclivi /usr/bin/snap[13857]: cmd.go:105: DEBUG: restarting into "/snap/core/curr 3月 12 09:10:28 jclivi multipassd[13857]: Unable to determine subnet for the mpqemubr0 subnet 3月 12 09:10:28 jclivi multipassd[13857]: Using AppArmor support lines 1-14...skipping... ● snap.multipass.multipassd.service - Service for snap application multipass.multipassd Loaded: loaded (/etc/systemd/system/snap.multipass.multipassd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-03-12 09:10:27 CST; 6min ago Main PID: 13857 (multipassd) Tasks: 9 (limit: 4915) Memory: 33.7M CPU: 1.590s CGroup: /system.slice/snap.multipass.multipassd.service └─13857 /snap/multipass/1784/bin/multipassd --verbosity debug --logger platform

3月 12 09:10:27 jclivi systemd[1]: Started Service for snap application multipass.multipassd. 3月 12 09:10:27 jclivi /usr/bin/snap[13857]: cmd.go:105: DEBUG: restarting into "/snap/core/current/usr/bin/snap" 3月 12 09:10:28 jclivi multipassd[13857]: Unable to determine subnet for the mpqemubr0 subnet 3月 12 09:10:28 jclivi multipassd[13857]: Using AppArmor support 3月 12 09:10:28 jclivi multipassd[13975]: Applying AppArmor policy: multipass.dnsmasq 3月 12 09:10:28 jclivi multipassd[13857]: 3月 12 09:10:28 jclivi multipassd[13857]: dnsmasq: 3月 12 09:10:28 jclivi multipassd[13857]: dnsmasq: cannot create DHCP socket: Permission denied 3月 12 09:10:28 jclivi multipassd[13857]: dnsmasq: cannot create DHCP socket: Permission denied

jclivi avatar Mar 12 '20 01:03 jclivi

@townsend2010 -- Logs begin at Thu 2020-03-12 01:40:13 CST, end at Thu 2020-03-12 09:27:22 CST. -- 3月 12 01:44:31 jclivi dnsmasq-dhcp[11424]: DHCPRELEASE(mpqemubr0) 10.84.117.100 52:54:00:aa:cc:57 3月 12 01:44:31 jclivi dnsmasq-dhcp[11424]: DHCPRELEASE(mpqemubr0) 10.84.117.252 52:54:00:6d:8a:18 3月 12 01:44:39 jclivi multipassd[10551]: QMP: {"timestamp": {"seconds": 1583948679, "microseconds": 531085}, "event": "NIC_RX_F 3月 12 01:44:42 jclivi dnsmasq-dhcp[11424]: DHCPDISCOVER(mpqemubr0) 52:54:00:5f:2f:85 3月 12 01:44:42 jclivi dnsmasq-dhcp[11424]: DHCPOFFER(mpqemubr0) 10.84.117.38 52:54:00:5f:2f:85 3月 12 01:44:42 jclivi dnsmasq-dhcp[11424]: DHCPDISCOVER(mpqemubr0) 52:54:00:5f:2f:85 3月 12 01:44:42 jclivi dnsmasq-dhcp[11424]: DHCPOFFER(mpqemubr0) 10.84.117.38 52:54:00:5f:2f:85 3月 12 01:44:42 jclivi dnsmasq-dhcp[11424]: DHCPREQUEST(mpqemubr0) 10.84.117.38 52:54:00:5f:2f:85 3月 12 01:44:42 jclivi dnsmasq-dhcp[11424]: DHCPACK(mpqemubr0) 10.84.117.38 52:54:00:5f:2f:85 ubuntu-18 3月 12 01:44:53 jclivi multipassd[9306]: Applying AppArmor policy: multipass.qemu-img 3月 12 01:44:53 jclivi multipassd[9317]: Applying AppArmor policy: multipass.qemu-img 3月 12 01:44:54 jclivi multipassd[10551]: process working dir '/snap/multipass/1784/qemu' 3月 12 01:44:54 jclivi multipassd[10551]: process program 'qemu-system-x86_64' 3月 12 01:44:54 jclivi multipassd[10551]: process arguments '--enable-kvm, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/ 3月 12 01:44:54 jclivi multipassd[10551]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx 3月 12 01:44:54 jclivi multipassd[9356]: Applying AppArmor policy: multipass.ubuntu.qemu-system-x86_64 3月 12 01:44:54 jclivi multipassd[10551]: process started 3月 12 01:44:54 jclivi multipassd[10551]: QMP: {"QMP": {"version": {"qemu": {"micro": 1, "minor": 11, "major": 2}, "package": "( 3月 12 01:44:55 jclivi multipassd[10551]: QMP: {"return": {}} 3月 12 01:45:13 jclivi multipassd[10551]: QMP: {"timestamp": {"seconds": 1583948713, "microseconds": 286961}, "event": "RTC_CHAN 3月 12 01:45:25 jclivi multipassd[10551]: failed to obtain exit status for remote process '[ -e /var/lib/cloud/instance/boot-fin

jclivi avatar Mar 12 '20 01:03 jclivi

@jclivi this is quite weird, can you try:

$ sudo systemctl edit snap.multipass.multipassd.service
# add the following lines and save the file
# [Service]
# Environment=DISABLE_APPARMOR=1
$ snap restart multipass

And then try again?

The journal logs should say AppArmor disabled by environment variable.

After modifying and restarting, I can mount the file to vm. jclivi@jclivi:~$ multipass mount /home/jclivi/data ubuntu:/home/ubuntu/data
jclivi@jclivi:~$ multipass shell ubuntu ubuntu@ubuntu:~$ cd data/ ubuntu@ubuntu:~/data$ ls ubuntu@ubuntu:~/data$ ls ubuntu@ubuntu:~/data$ ls 1.txt

jclivi avatar Mar 12 '20 01:03 jclivi

sudo systemctl status snap.multipass.* ● snap.multipass.multipassd.service - Service for snap application multipass.multipassd Loaded: loaded (/etc/systemd/system/snap.multipass.multipassd.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/snap.multipass.multipassd.service.d └─override.conf Active: active (running) since Thu 2020-03-12 09:22:42 CST; 17min ago Main PID: 22679 (multipassd) Tasks: 18 (limit: 4915) Memory: 809.6M CPU: 1min 44.713s CGroup: /system.slice/snap.multipass.multipassd.service ├─22679 /snap/multipass/1784/bin/multipassd --verbosity debug --logger platform ├─22764 /snap/multipass/1784/usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/snap/multipass/common ├─25099 /snap/multipass/1784/usr/bin/qemu-system-x86_64 --enable-kvm -device virtio-scsi-pci,id=scsi0 -drive file=/v └─26500 /snap/multipass/1784/bin/sshfs_server 10.84.117.74 22 ubuntu /home/jclivi/data /home/ubuntu/data 1000:-1, 10

3月 12 09:26:25 jclivi multipassd[22679]: mounting /home/jclivi/data => /home/ubuntu/data in ubuntu 3月 12 09:26:25 jclivi multipassd[22679]: process program '/snap/multipass/1784/bin/sshfs_server' 3月 12 09:26:25 jclivi multipassd[22679]: process arguments '10.84.117.74, 22, ubuntu, /home/jclivi/data, /home/ubuntu/data, 100 3月 12 09:26:26 jclivi multipassd[22679]: SSHFS was not found on the host: 10.84.117.74 3月 12 09:26:26 jclivi multipassd[22679]: Mount '/home/ubuntu/data' in instance "ubuntu" has stopped unexpectedly: Process retur 3月 12 09:26:26 jclivi multipassd[22679]: Installing sshfs in 'ubuntu' 3月 12 09:27:24 jclivi multipassd[22679]: mounting /home/jclivi/data => /home/ubuntu/data in ubuntu

jclivi avatar Mar 12 '20 01:03 jclivi

Let's keep this open as it is a real issue that needs fixing. The environment variable is just a work around.

townsend2010 avatar Mar 12 '20 12:03 townsend2010