multipass
multipass copied to clipboard
canonical
Error accessing VM after launch; state is always `unknown`
Describe the bug
After setting up a VM from the charm-dev blueprint using Multipass and successfully launching it as per the Juju tutorial, I am unable to access it.
To Reproduce I tried opening a shell into the instance like this: 1.
$ multipass shell my-juju-vm
shell failed: Cannot retrieve credentials in unknown state
The instance exists but in an unknown state: 2.
$ multipass list
Name State IPv4 Image
my-juju-vm Unknown -- Ubuntu 22.04 LTS
Expected behaviour Something like this:
$ multipass shell my-juju-vm
Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-100-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Mon Mar 18 17:11:59 CET 2024
System load: 0.0 Processes: 117
Usage of /: 5.6% of 28.89GB Users logged in: 1
Memory usage: 3% IPv4 address for ens3: 10.238.98.63
Swap usage: 0%
* Strictly confined Kubernetes makes edge and IoT secure. Learn how MicroK8s
just raised the bar for easy, resilient and secure K8s cluster deployment.
https://ubuntu.com/engage/secure-kubernetes-at-the-edge
Expanded Security Maintenance for Applications is not enabled.
16 updates can be applied immediately.
1 of these updates is a standard security update.
To see these additional updates run: apt list --upgradable
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
Last login: Mon Mar 18 16:09:16 2024 from 10.238.98.1
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
Logs
Ago 05 17:21:12 skoech systemd[1]: Started Service for snap application multipa>
Ago 05 17:21:13 skoech multipassd[83385]: Unable to determine subnet for the mp>
Ago 05 17:21:13 skoech multipassd[83385]: Using AppArmor support
Ago 05 17:21:13 skoech multipassd[83385]: Starting dnsmasq
Ago 05 17:21:13 skoech multipassd[83385]: Applied AppArmor policy: multipass.dn>
Ago 05 17:21:13 skoech multipassd[83385]: [83514] started: dnsmasq --keep-in-fo>
Ago 05 17:21:13 skoech dnsmasq[83514]: started, version 2.90 cachesize 150
Ago 05 17:21:13 skoech dnsmasq[83514]: compile time options: IPv6 GNU-getopt DB>
Ago 05 17:21:13 skoech dnsmasq-dhcp[83514]: DHCP, IP range 10.196.97.2 -- 10.19>
Ago 05 17:21:13 skoech dnsmasq-dhcp[83514]: DHCP, sockets bound exclusively to >
Ago 05 17:21:13 skoech dnsmasq[83514]: using only locally-known addresses for m>
Ago 05 17:21:13 skoech dnsmasq[83514]: reading /etc/resolv.conf
Ago 05 17:21:13 skoech dnsmasq[83514]: using nameserver 127.0.0.53#53
Ago 05 17:21:13 skoech dnsmasq[83514]: using only locally-known addresses for m>
Ago 05 17:21:13 skoech dnsmasq[83514]: read /etc/hosts - 8 names
Ago 05 17:21:13 skoech dnsmasq-dhcp[83514]: read /var/snap/multipass/common/dat>
Ago 05 17:21:13 skoech multipassd[83385]: [83515] started: iptables-nft --wait >
Ago 05 17:21:13 skoech multipassd[83385]: # Warning: iptables-legacy tables pre>
Ago 05 17:21:13 skoech multipassd[83385]: Using iptables-nft for firewall rules.
Ago 05 17:21:13 skoech multipassd[83385]: [83516] started: iptables-nft --wait >
Ago 05 17:21:13 skoech multipassd[83385]: # Warning: iptables-legacy tables pre>
Ago 05 17:21:13 skoech multipassd[83385]: [83517] started: iptables-nft --wait >
Ago 05 17:21:13 skoech multipassd[83385]: # Warning: iptables-legacy tables pre>
lines 1-23...skipping...
Ago 05 17:21:12 skoech systemd[1]: Started Service for snap application multipass.multipassd.
Ago 05 17:21:13 skoech multipassd[83385]: Unable to determine subnet for the mpqemubr0 subnet
Ago 05 17:21:13 skoech multipassd[83385]: Using AppArmor support
Ago 05 17:21:13 skoech multipassd[83385]: Starting dnsmasq
Ago 05 17:21:13 skoech multipassd[83385]: Applied AppArmor policy: multipass.dnsmasq
Ago 05 17:21:13 skoech multipassd[83385]: [83514] started: dnsmasq --keep-in-foreground --strict-order --bind-interfaces --pid-file --domain=multipass --local=/multipass/ --except-interface=lo --interface=mpq>
Ago 05 17:21:13 skoech dnsmasq[83514]: started, version 2.90 cachesize 150
Ago 05 17:21:13 skoech dnsmasq[83514]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset no-nftset auth cryptohash DNSSEC loop-detect inotify dumpfile
Ago 05 17:21:13 skoech dnsmasq-dhcp[83514]: DHCP, IP range 10.196.97.2 -- 10.196.97.254, lease time infinite
Ago 05 17:21:13 skoech dnsmasq-dhcp[83514]: DHCP, sockets bound exclusively to interface mpqemubr0
Ago 05 17:21:13 skoech dnsmasq[83514]: using only locally-known addresses for multipass
Ago 05 17:21:13 skoech dnsmasq[83514]: reading /etc/resolv.conf
Ago 05 17:21:13 skoech dnsmasq[83514]: using nameserver 127.0.0.53#53
Ago 05 17:21:13 skoech dnsmasq[83514]: using only locally-known addresses for multipass
Ago 05 17:21:13 skoech dnsmasq[83514]: read /etc/hosts - 8 names
Ago 05 17:21:13 skoech dnsmasq-dhcp[83514]: read /var/snap/multipass/common/data/multipassd/network/dnsmasq.hosts
Ago 05 17:21:13 skoech multipassd[83385]: [83515] started: iptables-nft --wait -t filter --list-rules
Ago 05 17:21:13 skoech multipassd[83385]: # Warning: iptables-legacy tables present, use iptables-legacy to see them
Ago 05 17:21:13 skoech multipassd[83385]: Using iptables-nft for firewall rules.
Ago 05 17:21:13 skoech multipassd[83385]: [83516] started: iptables-nft --wait -t filter --list-rules
Ago 05 17:21:13 skoech multipassd[83385]: # Warning: iptables-legacy tables present, use iptables-legacy to see them
Ago 05 17:21:13 skoech multipassd[83385]: [83517] started: iptables-nft --wait -t nat --list-rules
Ago 05 17:21:13 skoech multipassd[83385]: # Warning: iptables-legacy tables present, use iptables-legacy to see them
Ago 05 17:21:13 skoech multipassd[83385]: [83518] started: iptables-nft --wait -t mangle --list-rules
Ago 05 17:21:13 skoech multipassd[83385]: # Warning: iptables-legacy tables present, use iptables-legacy to see them
Ago 05 17:21:13 skoech multipassd[83385]: [83519] started: iptables-nft --wait -t raw --list-rules
Ago 05 17:21:13 skoech multipassd[83385]: # Warning: iptables-legacy tables present, use iptables-legacy to see them
Ago 05 17:21:13 skoech multipassd[83385]: [83520] started: iptables-nft --wait -t filter --insert INPUT --in-interface mpqemubr0 --protocol udp --dport 67 --jump ACCEPT --match comment --comment generated for>
Ago 05 17:21:13 skoech multipassd[83385]: [83521] started: iptables-nft --wait -t filter --insert INPUT --in-interface mpqemubr0 --protocol udp --dport 53 --jump ACCEPT --match comment --comment generated for>
Ago 05 17:21:13 skoech multipassd[83385]: [83522] started: iptables-nft --wait -t filter --insert INPUT --in-interface mpqemubr0 --protocol tcp --dport 53 --jump ACCEPT --match comment --comment generated for>
Ago 05 17:21:13 skoech multipassd[83385]: [83523] started: iptables-nft --wait -t filter --insert OUTPUT --out-interface mpqemubr0 --protocol udp --sport 67 --jump ACCEPT --match comment --comment generated f>
Ago 05 17:21:13 skoech multipassd[83385]: [83524] started: iptables-nft --wait -t filter --insert OUTPUT --out-interface mpqemubr0 --protocol udp --sport 53 --jump ACCEPT --match comment --comment generated f>
Ago 05 17:21:13 skoech multipassd[83385]: [83526] started: iptables-nft --wait -t filter --insert OUTPUT --out-interface mpqemubr0 --protocol tcp --sport 53 --jump ACCEPT --match comment --comment generated f>
Ago 05 17:21:14 skoech multipassd[83385]: [83527] started: iptables-nft --wait -t mangle --insert POSTROUTING --out-interface mpqemubr0 --protocol udp --dport 68 --jump CHECKSUM --checksum-fill --match commen>
Ago 05 17:21:14 skoech multipassd[83385]: [83531] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.196.97.0/24 --destination 224.0.0.0/24 --jump RETURN --match comment --comment generated >
Ago 05 17:21:14 skoech multipassd[83385]: [83532] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.196.97.0/24 --destination 255.255.255.255/32 --jump RETURN --match comment --comment gene>
Ago 05 17:21:14 skoech multipassd[83385]: [83533] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.196.97.0/24 ! --destination 10.196.97.0/24 --protocol tcp --jump MASQUERADE --to-ports 10>
Ago 05 17:21:14 skoech multipassd[83385]: [83534] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.196.97.0/24 ! --destination 10.196.97.0/24 --protocol udp --jump MASQUERADE --to-ports 10>
Ago 05 17:21:14 skoech multipassd[83385]: [83535] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.196.97.0/24 ! --destination 10.196.97.0/24 --jump MASQUERADE --match comment --comment ge>
Ago 05 17:21:14 skoech multipassd[83385]: [83536] started: iptables-nft --wait -t filter --insert FORWARD --destination 10.196.97.0/24 --out-interface mpqemubr0 --match conntrack --ctstate RELATED,ESTABLISHED>
Ago 05 17:21:14 skoech multipassd[83385]: [83537] started: iptables-nft --wait -t filter --insert FORWARD --source 10.196.97.0/24 --in-interface mpqemubr0 --jump ACCEPT --match comment --comment generated for>
Ago 05 17:21:14 skoech multipassd[83385]: [83538] started: iptables-nft --wait -t filter --insert FORWARD --in-interface mpqemubr0 --out-interface mpqemubr0 --jump ACCEPT --match comment --comment generated f>
Ago 05 17:21:14 skoech multipassd[83385]: [83539] started: iptables-nft --wait -t filter --append FORWARD --in-interface mpqemubr0 --jump REJECT --reject-with icmp-port-unreachable --match comment --comment g>
Ago 05 17:21:14 skoech multipassd[83385]: [83540] started: iptables-nft --wait -t filter --append FORWARD --out-interface mpqemubr0 --jump REJECT --reject-with icmp-port-unreachable --match comment --comment >
Ago 05 17:21:15 skoech multipassd[83385]: Loading "anbox-cloud-appliance" v1
Ago 05 17:21:15 skoech multipassd[83385]: Loading "charm-dev" v1
Ago 05 17:21:15 skoech multipassd[83385]: Loading "docker" v1
Ago 05 17:21:15 skoech multipassd[83385]: Loading "jellyfin" v1
Ago 05 17:21:15 skoech multipassd[83385]: Loading "minikube" v1
Ago 05 17:21:15 skoech multipassd[83385]: Loading "ros-noetic" v1
Ago 05 17:21:15 skoech multipassd[83385]: Loading "ros2-humble" v1
Ago 05 17:21:15 skoech multipassd[83385]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket
Ago 05 17:21:15 skoech multipassd[83385]: fetch manifest periodically
Ago 05 17:21:15 skoech multipassd[83385]: Starting Multipass 1.14.0
Ago 05 17:21:15 skoech multipassd[83385]: Daemon arguments: /snap/multipass/13042/bin/multipassd --verbosity debug --logger platform
lines 1-55
Additional info
- OS:
Ubuntu 22.04.4 LTS -
multipass version:1.14.0
$ multipass info
Name: my-juju-vm
State: Unknown
Snapshots: 0
IPv4: --
Release: --
Image hash: 123e55661336 (Ubuntu 22.04 LTS)
CPU(s): --
Load: --
Disk usage: --
Memory usage: --
Mounts: --
-
multipass get local.driver:qemu
Additional context I've tried these suggested fixes, unsuccessfully: #2316 #1268 #1358
@tmihoc is interested in this issue too.
I just performed a fresh install of Ubuntu 22.04 + installing multipass and also facing this issue.
cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
multipass --version
multipass 1.14.0
multipassd 1.14.0
multipass get local.driver
qemu
multipass launch --name "latest" -c 2 -m 6G --disk 20G
multipass info
Name: latest
State: Unknown
Snapshots: 0
IPv4: --
Release: --
Image hash: dd8b691b3f0d (Ubuntu 24.04 LTS)
CPU(s): --
Load: --
Disk usage: --
Memory usage: --
Mounts: --
Hi @skoech, sorry for the late reply. Would you be able to provide logs specifically around launching or starting a vm? The above logs are from just the Multipass daemon starting up and don't tell anything about the operation of the vm.
Just run the offending Multipass command and then check the log for new entries. Or, running the offending Multipass command with the -vvvv option will print some logs to stdout.
I have a similar problem, my log is as follows:
$ multipass start -vvvv ycc39
[2024-11-21T11:07:37.045] [trace] [daemon] Loading AppArmor policy:
#include <tunables/global>
profile multipass.ycc39.qemu-system-x86_64 flags=(attach_disconnected) {
#include <abstractions/base>
#include <abstractions/consoles>
#include <abstractions/nameservice>
# required for reading disk images
capability dac_override,
capability dac_read_search,
capability chown,
# Enables modifying of file ownership and permissions
capability fsetid,
capability fowner,
# needed to drop privileges
capability setgid,
capability setuid,
network inet stream,
network inet6 stream,
# Allow multipassd send qemu signals
signal (receive) peer=snap.multipass.multipassd,
/dev/net/tun rw,
/dev/kvm rw,
/dev/ptmx rw,
/dev/kqemu rw,
@{PROC}/*/status r,
# When qemu is signaled to terminate, it will read cmdline of signaling
# process for reporting purposes. Allowing read access to a process
# cmdline may leak sensitive information embedded in the cmdline.
@{PROC}/@{pid}/cmdline r,
# Per man(5) proc, the kernel enforces that a thread may
# only modify its comm value or those in its thread group.
owner @{PROC}/@{pid}/task/@{tid}/comm rw,
@{PROC}/sys/kernel/cap_last_cap r,
owner @{PROC}/*/auxv r,
@{PROC}/sys/vm/overcommit_memory r,
# access to firmware's etc (selectively chosen for multipass' usage)
/snap/multipass/13666/qemu/* r,
# for save and resume
/{usr/,}bin/dash rmix,
/{usr/,}bin/dd rmix,
/{usr/,}bin/cat rmix,
# to execute bridge helper
/snap/multipass/13666/bin/bridge_helper,
# for restore
/{usr/,}bin/bash rmix,
# for file-posix getting limits since 9103f1ce
/sys/devices/**/block/*/queue/max_segments r,
# for gathering information about available host resources
/sys/devices/system/cpu/ r,
/sys/devices/system/node/ r,
/sys/devices/system/node/node[0-9]*/meminfo r,
/sys/module/vhost/parameters/max_mem_regions r,
# binary and its libs
/snap/multipass/13666/usr/bin/qemu-system-x86_64 ixr,
/snap/multipass/13666/{,usr/}lib/{,@{multiarch}/}{,**/}*.so* rm,
# CLASSIC ONLY: need to specify required libs from core snap
/{,var/lib/snapd/}snap/core18/*/{,usr/}lib/@{multiarch}/{,**/}*.so* rm,
# Disk images
/var/snap/multipass/common/data/multipassd/vault/instances/ycc39/ubuntu-24.04-server-cloudimg-amd64.img rwk, # QCow2 filesystem image
/var/snap/multipass/common/data/multipassd/vault/instances/ycc39/cloud-init-config.iso rk, # cloud-init ISO
# allow full access just to user-specified mount directories on the host
}
[2024-11-21T11:07:37.045] [warning] [apparmor] Failed to load AppArmor policy
#include <tunables/global>
profile multipass.ycc39.qemu-system-x86_64 flags=(attach_disconnected) {
#include <abstractions/base>
#include <abstractions/consoles>
#include <abstractions/nameservice>
# required for reading disk images
capability dac_override,
capability dac_read_search,
capability chown,
# Enables modifying of file ownership and permissions
capability fsetid,
capability fowner,
# needed to drop privileges
capability setgid,
capability setuid,
network inet stream,
network inet6 stream,
# Allow multipassd send qemu signals
signal (receive) peer=snap.multipass.multipassd,
/dev/net/tun rw,
/dev/kvm rw,
/dev/ptmx rw,
/dev/kqemu rw,
@{PROC}/*/status r,
# When qemu is signaled to terminate, it will read cmdline of signaling
# process for reporting purposes. Allowing read access to a process
# cmdline may leak sensitive information embedded in the cmdline.
@{PROC}/@{pid}/cmdline r,
# Per man(5) proc, the kernel enforces that a thread may
# only modify its comm value or those in its thread group.
owner @{PROC}/@{pid}/task/@{tid}/comm rw,
@{PROC}/sys/kernel/cap_last_cap r,
owner @{PROC}/*/auxv r,
@{PROC}/sys/vm/overcommit_memory r,
# access to firmware's etc (selectively chosen for multipass' usage)
/snap/multipass/13666/qemu/* r,
# for save and resume
/{usr/,}bin/dash rmix,
/{usr/,}bin/dd rmix,
/{usr/,}bin/cat rmix,
# to execute bridge helper
/snap/multipass/13666/bin/bridge_helper,
# for restore
/{usr/,}bin/bash rmix,
# for file-posix getting limits since 9103f1ce
/sys/devices/**/block/*/queue/max_segments r,
# for gathering information about available host resources
/sys/devices/system/cpu/ r,
/sys/devices/system/node/ r,
/sys/devices/system/node/node[0-9]*/meminfo r,
/sys/module/vhost/parameters/max_mem_regions r,
# binary and its libs
/snap/multipass/13666/usr/bin/qemu-system-x86_64 ixr,
/snap/multipass/13666/{,usr/}lib/{,@{multiarch}/}{,**/}*.so* rm,
# CLASSIC ONLY: need to specify required libs from core snap
/{,var/lib/snapd/}snap/core18/*/{,usr/}lib/@{multiarch}/{,**/}*.so* rm,
# Disk images
/var/snap/multipass/common/data/multipassd/vault/instances/ycc39/ubuntu-24.04-server-cloudimg-amd64.img rwk, # QCow2 filesystem image
/var/snap/multipass/common/data/multipassd/vault/instances/ycc39/cloud-init-config.iso rk, # cloud-init ISO
# allow full access just to user-specified mount directories on the host
}
: errno=10 ()
[2024-11-21T11:07:37.046] [debug] [ycc39] process working dir '/snap/multipass/13666/qemu'
[2024-11-21T11:07:37.046] [info] [ycc39] process program 'qemu-system-x86_64'
[2024-11-21T11:07:37.046] [info] [ycc39] process arguments '-bios, OVMF.fd, --enable-kvm, -cpu, host, -nic, tap,ifname=tap-ef2b6d13241,script=no,downscript=no,model=virtio-net-pci,mac=52:54:00:5c:d4:f7, -nic, bridge,br=br-multipass,model=virtio-net-pci,mac=02:ab:cd:ef:00:01,helper=/snap/multipass/13666/bin/bridge_helper, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/snap/multipass/common/data/multipassd/vault/instances/ycc39/ubuntu-24.04-server-cloudimg-amd64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 2, -m, 2048M, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/snap/multipass/common/data/multipassd/vault/instances/ycc39/cloud-init-config.iso'
[2024-11-21T11:07:37.051] [debug] [qemu-system-x86_64] [78912] started: qemu-system-x86_64 -nographic -dump-vmstate /tmp/multipassd.zHBZea
[2024-11-21T11:07:37.100] [info] [ycc39] process state changed to Starting
[2024-11-21T11:07:37.109] [info] [ycc39] process state changed to Running
[2024-11-21T11:07:37.109] [debug] [qemu-system-x86_64] [78916] started: qemu-system-x86_64 -bios OVMF.fd --enable-kvm -cpu host -nic tap,ifname=tap-ef2b6d13241,script=no,downscript=no,model=virtio-net-pci,mac=52:54:00:5c:d4:f7 -nic bridge,br=br-multipass,model=virtio-net-pci,mac=02:ab:cd:ef:00:01,helper=/snap/multipass/13666/bin/bridge_helper -device virtio-scsi-pci,id=scsi0 -drive file=/var/snap/multipass/common/data/multipassd/vault/instances/ycc39/ubuntu-24.04-server-cloudimg-amd64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 2 -m 2048M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/snap/multipass/common/data/multipassd/vault/instances/ycc39/cloud-init-config.iso
[2024-11-21T11:07:37.109] [info] [ycc39] process started
start failed: The following errors occurred:
ycc39: timed out waiting for response
The odd thing is, I can ssh into the VM just find, it's only Multipass that can't detect the state.
Hi @yinchi, would you be able to open a new GitHub issue with all the specified information? We would be able to better diagnose your issue that way.
Hi @sharder996, created #3790 (details may be slightly different as I ran everything again).