Trouble connecting to VM on start

[ryanhanks-bestow]

trafficstars

Your Question

I read through the recommended steps for creating a new VM from scratch, but didn't feel I needed that level of customization in my image over what tart clone ghcr.io/cirruslabs/macos-sonoma-xcode:latest runner would offer.

Is there a reason using this as a starting point would cause me issues? I supply tartelet with the proper username / password for the vm but get stuck in a loop of failed attempts to connect to the machine:

15:17:01.227 INFO: Start virtual machine named runner-1
15:17:14.189 INFO: The GitHub app has not been installed. Please install it from the developer settings.
15:17:14.190 INFO: Could not connect to the virtual machine over SSH, so the virtual machine will be shut down.
15:17:14.193 INFO: Virtual machine named runner-1 stopped with message: Failed establishing SSH connection
15:17:14.525 INFO: Start virtual machine named runner-1
15:17:24.629 INFO: The GitHub app has not been installed. Please install it from the developer settings.
15:17:24.633 INFO: Could not connect to the virtual machine over SSH, so the virtual machine will be shut down.
15:17:24.638 INFO: Virtual machine named runner-1 stopped with message: Failed establishing SSH connection

I confirm the machine when booted with tart run will accept my ssh logins using the same username/password I'm giving tartelet (admin/admin), and I believe my github app and tartelet github settings are correctly configured and supplied with the proper key...

Anyone know what I'm running into here? Are both of the errors I see in these logs potential culprits? Or is one related to the other?

Did I miss something in the VM setup instructions that I need to do to this VM? Or is there a reason it's not a good fit for tartelet?

Thanks.

https://github.com/user-attachments/assets/ba3505f4-f8a7-4b7b-a1f7-a6836bbfb49d

(this screen grab was taken after playing around in the VM a bit, but I originally produced this same behavior on a pristine clone)

Any additional context?

No response

[krzysztof-pawlicki-nemo]

it behaves similarly after yesterday's upgrade to macOS sequoia on host. I was debugging the app a bit and seems like:

• Tartelet (through tart) can not get ip address of guest machine
• After removing all the credentials from keychain (I thought that maybe there is some strange cache and wanted to get to starting point like fresh install) it appears that I can not write to the keychain (OSStatus [-34018]). Entitlements are updated to match my bundle id.

[nmichlo]

Same issue here after upgrading to sequoia

[krzysztof-pawlicki-nemo]

I've spent a bit more time on trying to figure out what's behind and so far seems like there is a new Local Network Privacy feature (https://forums.developer.apple.com/forums/thread/760964) which probably causes the issue.

• temporarily I've removed storing of the SecKey and am loading it directly from filesystem. Password items can be successfully stored/read in the keychain.
• in my previous comment I've stated that ip address of guest can not be obtain. Apparently this ain't true cause even though respective message is logged in console the app get's the proper IP address.
• there is an error thrown in CitadelSSHClient on the connect attempt. At the same time I can SSH from terminal.
• so far for a test I've tried launching the app from Xcode as a root (which is supposed to bypass local network privacy limits) but then tart images are not listed.

[ryanhanks-bestow]

@krzysztof-pawlicki-nemo in regards to the test you mentioned, I assume you're probably aware, but in case you missed it, I believe the list of VMs by default pulls from ~/.tart and is configurable via TART_HOME

[ryanhanks-bestow]

FWIW, I reproduced this again on a fresh macOS install (14.7). Steps to reproduce:

1. Clean macOS install
2. install brew
3. install tart
4. clone ghcr.io/cirruslabs/macos-sonoma-xcode:latest
5. Install tartelet
6. configure tartelet

Also, I experienced this with the prior version of tartelet.

If anyone knows the max supported macOS (or can confirm a working version), please let me know.

[ryanhanks-bestow]

Noticed some stuff in the diff of https://github.com/shapehq/tartelet/releases/tag/0.8.1 that looked possibly related to what I saw in the logs and what @krzysztof-pawlicki-nemo was digging into so I tried https://github.com/shapehq/tartelet/releases/tag/0.8.0 and looks like this version is working (at least doesn't cause the restart-loop I was seeing in in 0.10.0 and 0.9.0). I'll play around with it a bit later to see if I can get some workflows running and report back.

cc @nmichlo

[ryanhanks-bestow]

I got this running workflow with the 0.8.0 release. With this release I was able to see that the failure I was experiencing was indeed inherent to the specifics of ghcr.io/cirruslabs/macos-sonoma-xcode.

The clone of ghcr.io/cirruslabs/macos-sonoma-xcode comes with lots of nice things – one of those happens to be a preinstallation of actions-runner in ~/. This conflicts with the boot.sh script and causes it to fail.

Editing the virtual machine and removing ~/actions-runner allowed the machine to fully boot and register, and I've been successfully running workflows.

This problem seems to be what was also ailing my tests on 0.9.0 and 0.10.0, which are also now working for me. The 0.8.0 release just happened to give me a bit more visibility into what was happening.

In summary, to use ghcr.io/cirruslabs/macos-sonoma-xcode:

1. tart clone ghcr.io/cirruslabs/macos-sonoma-xcode:latest runner
2. Configure tartelet
3. Edit the Virtual Machine and:

ssh github.com # this adds github as a known host
rm -rf ~/actions-runner # removes conflicting actions-runner installation

4. Shutdown the vm
5. Start tartelet

Again, I'm running this on Sonoma 14.7 metal, so the issues @krzysztof-pawlicki-nemo and @nmichlo chimed in with may or may not be related. I never confirmed – are y'all also using some variant of ghcr.io/cirruslabs/macos-sonoma-xcode?

[krzysztof-pawlicki-nemo]

Hey @ryanhanks-bestow , thanks for your input. I was busy with other tasks and put it on a side for some time but now can digg again. I use locally build (via packer) tart image. This is a completely fresh install from macOS Sonoma image without GitHub Actions preinstall. So far the very same image was used as a custom GitLab runner but I'd like to migrate to self hosted GitHub. I wanted to do small audit of the code and compiled/notarised Tartelet on my own. The same Tartelet binary works on different host (Sonoma) with the very same tart image. The issue appeared after I upgraded my dev machine (working as additional runner - host) to Sequoia.

Btw. this project looks very promising, thank you @simonbs for all the nice work you've put into this! I hope my feasibility phase will end with a success on Sequoia host and I'll be able to contribute bit more in here.

[kuhnroyal]

I had this happen while investigating #93

Pretty sure this is related to the permissions in "Security & Privacy -> Local Network" - if you have multiple versions of Tartlet installed (custom builds with different Bundle ID), Tartlet will show up multiple times as enabled.

However, the newer versions do not ask again for the permission and also do not have the previously granted permission, because it is technically a different app/bundle. So the secondary Tartlet version crashes when trying to scan the network. Some parts of MacOS think that the permission is already granted, including the UI. This happens for all apps with the same name.

See https://forums.developer.apple.com/forums/thread/766270

Workaround: Reset the Tartlet permissions in "Security & Privacy -> Local Network".

[krzysztof-pawlicki-nemo]

So good to be back from holiday and seeing your post ;) Thank you @kuhnroyal, that was it. The thread on dev forum was also interesting read.

[krzysztof-pawlicki-nemo]

@ryanhanks-bestow going back to your initial post I think it's also worth double checking DHCP lease on the host machine. At some point I've hit the problem of all the ip addresses in range used by tart being used. The VM started to get ip from local subnet assigned but it could not communicate with host. If I remember correctly https://tart.run/faq/#changing-the-default-dhcp-lease-time and https://github.com/cirruslabs/tart/issues/657#issuecomment-1812129793 was helpful.

[simonbs]

@ryanhanks-bestow going back to your initial post I think it's also worth double checking DHCP lease on the host machine. At some point I've hit the problem of all the ip addresses in range used by tart being used. The VM started to get ip from local subnet assigned but it could not communicate with host. If I remember correctly https://tart.run/faq/#changing-the-default-dhcp-lease-time and cirruslabs/tart#657 (comment) was helpful.

Indeed. Just wanted to highlight that's also briefly covered in Tartelet's wiki: https://github.com/shapehq/tartelet/wiki/Setting-Up-a-Host-Machine#13-reduce-dhcp-lease-time

[jsorge]

I'm setting up a new M4 Mac mini running macOS 15.1 on both the runner and the VM, and have followed all the directions in the wiki but when Tartelet starts up the VM it does just what was happening on this thread too. The VM starts up and then shuts down without the runner script running. I think it's failing at the ssh step but I don't know that for sure. I've checked the host logs and nothing especially helpful is sticking out. Could this be a problem with 15.1 in the VM? (I have a fleet of M4s running 15.1 as their host OS but still running Sonoma in VMs).

[hisaac]

I initially hit this issue after updating Tartelet, but only on one of our CI machines. Through some random troubleshooting, I eventually resolved it by:

1. Deleting Tartelet.app.
2. Downloading an older version of Tartelet.app (specifically for me I downloaded https://github.com/shapehq/tartelet/releases/tag/0.10.0, but I'm not sure if that matters).
3. Launch that older version of the app, and observe that things work as expected.
4. Downloaded the newest version of Tartelet.app.
5. Launched that version, and observe that it was now working as expected.

Not sure why this fixed it, or if it will fix it for anyone else, but wanted to share just in case.

[jsorge]

Aha I found it after looking at the Tartelet logs (thanks @hisaac for pointing me to that location in Application Support) and found that the GitHub App I created wasn't yet installed in my org. Doing that let the VM fire up right away (though I'm surprised that ssh-ing in to the VM is tied to the app being available on GitHub).

But I'm all good now!

[niw]

Here is my findings about this problem.

1. In some situation, looks like Tartelet app seems failing to SSH to any virtual machines, may be somehow it losts a permission to access Local Network. To me, it was happening after forcible Mac restarts due to power outage. Toggling the switch for Tartelet in Privacy settings in Settings seems workaround this situation.
2. If it failed to SSH, it keeps restarting virtual machine and keep assinging new IP address by DHCP and thus the IP address will be running out. If it's happening, from that point, even if Tartelet can access Local Network, it can’t SSH due to virtual machine can’t get IP address anymore.

• To identify this situation, look at the log file in ~/Library/Application Support/dk.shape.Tartelet/Logs.
• To mitigate the DHCP lease issue, reducing DHCP lease time described by @krzysztof-pawlicki-nemo’s link may help, but once Tartelet keeps restarting the virtual machine for a while, it may not help because anyways it will run out the IP addresses.

[Gussy]

Some additional ideas for anyone running into this:

1. The GitHub App username/org is case-sensitive
2. Runner tags can't have spaces, must be like tag1,tag2,tag3 not tag1, tag2, tag3