multipass icon indicating copy to clipboard operation
multipass copied to clipboard

SSH sessions getting dropped on macOS

Open ricab opened this issue 1 year ago • 9 comments

Multipass also became unusable for me after upgrading to 1.15.0 on Sequoia M1.

First any SSH session to VM will be lost after some time, ranging from seconds to 1-2 mins.

The logs is filled infinitely with

[2025-03-05T22:06:34.233] [info] [enduring-coonhound] SSH session disconnected [2025-03-05T22:06:34.233] [debug] [enduring-coonhound] Caching new SSH session [2025-03-05T22:06:36.214] [info] [enduring-coonhound] SSH session disconnected [2025-03-05T22:06:36.214] [debug] [enduring-coonhound] Caching new SSH session [2025-03-05T22:06:38.221] [info] [enduring-coonhound] SSH session disconnected [2025-03-05T22:06:38.222] [debug] [enduring-coonhound] Caching new SSH session [2025-03-05T22:06:40.224] [info] [enduring-coonhound] SSH session disconnected [2025-03-05T22:06:40.224] [debug] [enduring-coonhound] Caching new SSH session [2025-03-05T22:06:42.232] [info] [enduring-coonhound] SSH session disconnected [2025-03-05T22:06:42.233] [debug] [enduring-coonhound] Caching new SSH session [2025-03-05T22:06:44.239] [info] [enduring-coonhound] SSH session disconnected

and the UI keeps showing "waiting for demon ..."

I'm thinking go back to try VirtualBox now ...

Originally posted by @dohoangkhiem in #3721

ricab avatar Mar 05 '25 16:03 ricab

Hi @dohoangkhiem, your case appears to be different from the other one. Would you mind providing a few more details?

  • logs (Please provide logs from the daemon, see accessing logs on where to find them on your platform.)
  • any steps you to reproduce the issue?
  • macOS version
  • multipass get local.driver
  • have you tried https://github.com/canonical/multipass/releases/tag/v1.15.1 ?

ricab avatar Mar 05 '25 16:03 ricab

Sorry for jumping in on this issue, but I'm experiencing it too, on MacOS Sequoia 15.3.1.

 % sudo tail -10  /Library/Logs/Multipass/multipassd.log
[2025-03-06T15:28:31.674] [info] [primary] SSH session disconnected
[2025-03-06T15:28:31.674] [debug] [primary] Caching new SSH session
[2025-03-06T15:28:36.820] [info] [primary] SSH session disconnected
[2025-03-06T15:28:36.821] [debug] [primary] Caching new SSH session
[2025-03-06T15:28:42.016] [info] [primary] SSH session disconnected
[2025-03-06T15:28:42.016] [debug] [primary] Caching new SSH session
[2025-03-06T15:28:47.207] [info] [primary] SSH session disconnected
[2025-03-06T15:28:47.207] [debug] [primary] Caching new SSH session
[2025-03-06T15:28:52.349] [info] [primary] SSH session disconnected
[2025-03-06T15:28:52.349] [debug] [primary] Caching new SSH session

It requires no steps to make this happen. I restart my Mac, try to access a Multipass instance, and my connection is dropped very shortly after connecting (less than 30 s). After that, I cannot establish a new ssh connection, and 'multipass shell ' also fails.

Multipass info:

% multipass version
multipass   1.15.1+mac
multipassd  1.15.1+mac
% multipass get local.driver
qemu

I am in effect completely blocked from using Multipass at the moment.

kedmison avatar Mar 06 '25 20:03 kedmison

Hey, @kedmison! Take a look at this comment, perhaps you are in the same situation https://github.com/canonical/multipass/issues/3864#issuecomment-2585132672

andrei-toterman avatar Mar 07 '25 13:03 andrei-toterman

I checked it and iTerm local network access is already enabled. (I use iTerm pretty much always) I also checked Terminal.app and can't ssh via it.

I also tried Multipass.app, and am getting 'waiting for daemon' when trying to view the list of instances.

Which MacOS extensions are involved/required for Multipass? I was trying to clean up some very old system extensions (that have been carried forward through a decade of upgrades). Maybe I've done something that is causing this?

kedmison avatar Mar 07 '25 14:03 kedmison

Sorry for jumping in on this issue, but I'm experiencing it too, on MacOS Sequoia 15.3.1.

 % sudo tail -10  /Library/Logs/Multipass/multipassd.log
[2025-03-06T15:28:31.674] [info] [primary] SSH session disconnected
[2025-03-06T15:28:31.674] [debug] [primary] Caching new SSH session
[2025-03-06T15:28:36.820] [info] [primary] SSH session disconnected
[2025-03-06T15:28:36.821] [debug] [primary] Caching new SSH session
[2025-03-06T15:28:42.016] [info] [primary] SSH session disconnected
[2025-03-06T15:28:42.016] [debug] [primary] Caching new SSH session
[2025-03-06T15:28:47.207] [info] [primary] SSH session disconnected
[2025-03-06T15:28:47.207] [debug] [primary] Caching new SSH session
[2025-03-06T15:28:52.349] [info] [primary] SSH session disconnected
[2025-03-06T15:28:52.349] [debug] [primary] Caching new SSH session

Unrelated to the Mac, but I did observe a similar behavior on Windows while working on a new feature. In my case, the daemon was assuming that the VM was running but in reality, the VM was turned off gracefully by me, without Multipass knowing about it. Hence, the daemon constantly tries to reestablish the connection. Posting this thinking it might be relevant.

xmkg avatar Mar 24 '25 11:03 xmkg

Interesting, so how could a new SSH session be established and cached? 🤔 Perhaps the issue is Multipass thinking it established a connection when it couldn't...

ricab avatar Mar 24 '25 11:03 ricab

so how could a new SSH session be established and cached

The "caching new" log is printed before the connection attempt, regardless of the result:

https://github.com/canonical/multipass/blob/f7ba74f83f67677a749b56b8b8602df0c35f10e1/src/platform/backends/shared/base_virtual_machine.cpp#L272-L276

The daemon is busy-looping the renew_ssh_session(), assuming that it would be able to reach the VM. The question is, is the VM really in a reachable state but it couldn't, or daemon is unaware that the VM is in an unreachable state.

xmkg avatar Mar 24 '25 11:03 xmkg

Ah right, we should verify the VM's status by other means before retrying.

ricab avatar Mar 24 '25 12:03 ricab

Same issue with version 1.15.1+mac

sudo tail -10  /Library/Logs/Multipass/multipassd.log
[2025-05-13T23:00:17.801] [info] [reliable-sprat] SSH session disconnected
[2025-05-13T23:00:17.801] [debug] [reliable-sprat] Caching new SSH session
[2025-05-13T23:00:22.978] [info] [reliable-sprat] SSH session disconnected
multipass get local.driver
qemu

macOS Sequoia 15.4.1

gianpaj avatar May 13 '25 21:05 gianpaj

Hey @xmkg, do you remember this one? Didn't it get fixed?

ricab avatar Sep 12 '25 18:09 ricab

Hey @xmkg, do you remember this one? Didn't it get fixed?

Yeah, I remember -- I think we haven't fixed this one yet.

xmkg avatar Sep 12 '25 19:09 xmkg

Hey guys, anything new about this?

Mac OS Tahoe 26.0.1 Multipass Version 1.16.1

[2025-10-16T09:28:05.598] [info] [craft4] SSH session disconnected
[2025-10-16T09:28:05.598] [debug] [craft4] Caching new SSH session
[2025-10-16T09:28:07.605] [info] [craft4] SSH session disconnected
[2025-10-16T09:28:07.606] [debug] [craft4] Caching new SSH session
[2025-10-16T09:28:09.606] [info] [craft4] SSH session disconnected
[2025-10-16T09:28:09.606] [debug] [craft4] Caching new SSH session
[2025-10-16T09:28:11.612] [info] [craft4] SSH session disconnected
[2025-10-16T09:28:11.612] [debug] [craft4] Caching new SSH session
[2025-10-16T09:28:13.617] [info] [craft4] SSH session disconnected
[2025-10-16T09:28:13.617] [debug] [craft4] Caching new SSH session
[2025-10-16T09:28:15.621] [info] [craft4] SSH session disconnected
[2025-10-16T09:28:15.621] [debug] [craft4] Caching new SSH session

michalbulla avatar Oct 16 '25 07:10 michalbulla

This is still an issue (and also effects Windows), I could reproduce by following the Set up a graphical interface guide.

Sploder12 avatar Oct 17 '25 16:10 Sploder12