[BUG] crc setup fails on macOS Ventura (Intel & M1)

[What-Do-I-Know]

General information

• OS: macOS
• Hypervisor: vfkit
• Did you run crc setup before starting it (Yes/No)? Yes
• Running CRC on: Laptop

CRC version

OpenShift version: 4.11.18
Podman version: 4.2.0

CRC status

DEBU OpenShift version: 4.11.18
DEBU Podman version: 4.2.0
DEBU Running 'crc status'
DEBU Running 'sw_vers -productVersion'
DEBU sending request - Post "https://api.segment.io/v1/batch": dial tcp 0.0.0.0:443: connect: connection refused
DEBU 1 messages dropped because they failed to be sent and the client was closed
crc does not seem to be setup correctly, have you run 'crc setup'?

CRC config

Host Operating System

ProductVersion:		13.1
BuildVersion:		22C65

Steps to reproduce

1. Install CRC
2. UI wizard fails
3. Running crc setup manually does not succeed with the above mentioned error

Expected

crc setup completes with success.

Actual

crc setup fails with error: "Temporary error: daemon is not running yet (x8)"

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
Machine does not exist. Use 'crc start' to create it
$ crc cleanup
INFO Unloading and removing the daemon plist file
INFO Removing /etc/resolver/testing file
INFO Stopping CRC vfkit process
INFO Removing hosts file records added by CRC
INFO Removing pull secret from the keyring
INFO Removing older logs
INFO Removing CRC Machine Instance directory
INFO Removing crc executable symlink
Cleanup finished

$ crc stetup --log-level debug

DEBU CRC version: 2.12.0+ea98bb41                 
DEBU OpenShift version: 4.11.18                   
DEBU Podman version: 4.2.0                        
DEBU Running 'crc setup'                          
INFO Using bundle path /Users/jp/.crc/cache/crc_vfkit_4.11.18_arm64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking for obsolete admin-helper executable 
DEBU Checking if an older admin-helper executable is installed 
DEBU No older admin-helper executable found       
INFO Checking if running on a supported CPU architecture 
DEBU GOARCH is arm64 GOOS is darwin               
INFO Checking minimum RAM requirements            
DEBU Total memory of system is 17179869184 bytes  
INFO Checking if crc executable symlink exists    
INFO Checking if running emulated on a M1 CPU     
INFO Checking if vfkit is installed               
INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /Users/jp/.crc/cache/crc_vfkit_4.11.18_arm64.crcbundle exists 
DEBU /Users/jp/.crc/cache/crc_vfkit_4.11.18_arm64.crcbundle exists 
INFO Checking if old launchd config for tray and/or daemon exists 
INFO Checking if crc daemon plist file is present and loaded 
DEBU Running 'bash -c launchctl list | grep com.redhat.crc.daemon | awk '{print $1}'' 
DEBU launchd agent 'com.redhat.crc.daemon' is not running 
INFO Adding crc daemon plist file and loading it  
DEBU Creating plist for com.redhat.crc.daemon     
DEBU Running 'launchctl load -w /Users/jp/Library/LaunchAgents/com.redhat.crc.daemon.plist' 
DEBU Running 'launchctl stop com.redhat.crc.daemon' 
DEBU Running 'launchctl start com.redhat.crc.daemon' 
DEBU retry loop: attempt 0                        
DEBU error: Temporary error: daemon is not running yet - sleeping 2s 
DEBU retry loop: attempt 1                        
DEBU error: Temporary error: daemon is not running yet - sleeping 2s 
DEBU retry loop: attempt 2                        
DEBU error: Temporary error: daemon is not running yet - sleeping 2s 
DEBU retry loop: attempt 3                        
DEBU error: Temporary error: daemon is not running yet - sleeping 2s 
DEBU retry loop: attempt 4                        
DEBU error: Temporary error: daemon is not running yet - sleeping 2s 
DEBU retry loop: attempt 5                        
DEBU error: Temporary error: daemon is not running yet - sleeping 2s 
DEBU retry loop: attempt 6                        
DEBU error: Temporary error: daemon is not running yet - sleeping 2s 
DEBU retry loop: attempt 7                        
DEBU error: Temporary error: daemon is not running yet - sleeping 2s 
DEBU RetryAfter timeout after 8 tries             
DEBU Running 'sw_vers -productVersion'            
DEBU sending request - Post "https://api.segment.io/v1/batch": dial tcp 0.0.0.0:443: connect: connection refused 
DEBU 1 messages dropped because they failed to be sent and the client was closed 
Temporary error: daemon is not running yet (x8)

[cfergeau]

Can you try these commands manually?

$ ps aux |grep crc
$ launchctl list | grep com.redhat.crc.daemon
$ launchctl stop com.redhat.crc.daemon
$ launchctl load -w /Users/jp/Library/LaunchAgents/com.redhat.crc.daemon.plist

and then check again ps aux to see if thedaemon is running?

Can you also run ~/.crc/bin/crc daemon --log-level debug by hand?

[cfergeau]

Had a similar issue during unrelated testing, crc cleanup && crc setup helped in my case, I see this was not helpful for you :(

[What-Do-I-Know]

Checked the daemon was not working, then: launchctl load -w /Users/jp/Library/LaunchAgents/com.redhat.crc.daemon.plist launchctl list | grep com.redhat.crc.daemon 2829 0 com.redhat.crc.daemon

Then I can crc start

[What-Do-I-Know]

Stopping it and tunning by hand works (I also tested I can crc start when runnig the daemon manually):

~/.crc/bin/crc daemon --log-level debug    
DEBU CRC version: 2.12.0+ea98bb41                 
DEBU OpenShift version: 4.11.18                   
DEBU Podman version: 4.2.0                        
DEBU Running 'crc daemon'                         
INFO listening /Users/jp/.crc/tap.sock            
INFO listening /Users/jp/.crc/crc-http.sock       
0B sent to the VM, 0B received from the VM
0B sent to the VM, 0B received from the VM
0B sent to the VM, 0B received from the VM
0B sent to the VM, 0B received from the VM
0B sent to the VM, 0B received from the VM
0B sent to the VM, 0B received from the VM

Question is why the installer fails to start the daemon?

[gbraad]

Question is why the installer fails to start the daemon?

It will be hard to retry this now. Perhaps it would be best to describe this as part of a troubleshooting to prevent similar issues for users. Although, a crc cleanup && crc setup would have been cleaner to peform these steps.

[praveenkumar]

@What-Do-I-Know When it fails as part of crc setup can you open the console application and then go to Log Reports and then select launchd.log , check if you find out some more info around why it is failing as part of setup but works when you manually load it.

[cfergeau]

I've been able to reproduce something similar, but no idea if it's the same as your issue. I had a pop up open on my mac asking me to unlock a keychain, and this apparently is causing issues with starting the daemon through launchctl. The daemon is running but not answering to requests on its socket, crc start complains with Is 'crc daemon' running? Cannot reach daemon API: Get "http://unix/api/version": dial unix /Users/teuf/.crc/crc-http.sock: connect: connection refused If I retry crc cleanup && crc setup, the 'keychain unlock' popup shows again, and starting the daemon through launchctl fails again.

As soon as I closed all the keychain unlock popups, crc start proceeded as usual.

This started happening with commit 7a5d10aacd6ae441e5000d0a0914fade7d39f6a3

@anjannath

[anjannath]

Ahh :( , the "unlock keychain" pop-up comes from this part of the code, when we are initialising the new secret store https://github.com/crc-org/crc/blob/76d07a985c9a7be95beba9b9c796760b6c258817/pkg/crc/config/secret_config.go#L27-L32

It checks if the keychain can be used/accessible early on when initialising the config by trying to store some value, this was added to handle an error on Linux we observed on the linux CI where the gnome secret-service was not accessible and that resulted in some warning, which is still better as that happens when the crc config set|unset|get commands are ran but the current situation prevents the daemon for running on macOS when the Login keychain is locked.

while on a default macOS installation the Login keychain is unlocked once at login and remains unlocked for the whole session, in your case @cfergeau and also @What-Do-I-Know maybe it gets locked after sometime? and that is the reason the daemon is not able to start.

another reason that @adrianriobo mentioned on slack is if we are accessing via ssh but the GUI login was not done in that case launchd service does not start or keychain needs to be unlocked manually as the pop-up is only shown when a graphical login session is available.

as a workaround set the following settings (untick the "lock after" or "lock on sleep") for the Login keychain, open "keychain settings" by right-clicking on the login keychain..

or use the commands below if on ssh:

$ security list-keychains # find out the login keychains path
    "/Users/anjan/Library/Keychains/login.keychain-db"
    "/Library/Keychains/System.keychain"
$ security unlock-keychain /Users/anjan/Library/Keychains/login.keychain-db
password to unlock /Users/anjan/Library/Keychains/login.keychain-db:

I'll check if keychainAccessible can be removed or a replacement is needed to make the same check without having to try to store something

[anjannath]

@What-Do-I-Know did you try the workaround mentioned in https://github.com/crc-org/crc/issues/3458#issuecomment-1359982769 and does it help? are you running crc setup from an ssh session?