minikube start (vfkit) fails on "managed" M1 Mac

[adietish]

What Happened?

1. ASSERT: use a managed M1 mac where you cannot change the firewall
2. ASSERT: delete or other cleanup of prior minikube instances
3. EXEC: minikube start --driver=vfkit

Attach the log file

😄  minikube v1.36.0 on Darwin 15.5 (arm64)
    ▪ MINIKUBE_HOME=/Volumes/T7/.minikube
✨  Using the vfkit driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔑  Your firewall is blocking bootpd which is required for this configuration. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd


Password:🔥  Deleting "minikube" in vfkit ...
🤦  StartHost failed, but will try again: creating host: create host timed out in 360.000000 seconds
🔥  Creating vfkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔑  Your firewall is blocking bootpd which is required for this configuration. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd


😿  Failed to start vfkit VM. Running "minikube delete" may fix it: creating host: create host timed out in 360.000000 seconds

❌  Exiting due to DRV_CREATE_TIMEOUT: Failed to start host: creating host: create host timed out in 360.000000 seconds
💡  Suggestion: Try 'minikube delete', and disable any conflicting VPN or firewall software
🍿  Related issue: https://github.com/kubernetes/minikube/issues/7072

If I run the firewall modification manually, I am told that I cannot change the firewall:

❯ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
Password:
Firewall settings cannot be modified from command line on managed Mac computers.

Operating System

macOS (Default)

Driver

vfkit

here are the logs: https://gist.github.com/adietish/ee1d996a8fecd4f739825d0793dcbf3f

[afbjorklund]

You should be able to use "qemu" driver, I think both vfkit and krunkit have some privileged parts (like libvirt)?

If not for the firewall (like here), then for setting up the vmnet networking. While qemu has the user networking.

[adietish]

@afbjorklund thx for the pointer. I had tried this with a similar failure. Interestingly minikube tries to change the firewall even with the qemu driver and it then thinks that it could change it but actually failed to do so.

❯ minikube --driver=qemu start
😄  minikube v1.36.0 on Darwin 15.6 (arm64)
    ▪ MINIKUBE_HOME=/Volumes/T7/.minikube
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating qemu2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔑  Your firewall is blocking bootpd which is required for this configuration. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd


🔄  Successfully unblocked bootpd process from firewall, retrying
🔥  Deleting "minikube" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for 82:7d:5c:51:7c:e2
🔥  Creating qemu2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔑  Your firewall is blocking bootpd which is required for this configuration. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd


🔄  Successfully unblocked bootpd process from firewall, retrying
😿  Failed to start qemu2 VM. Running "minikube delete" may fix it: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for 9a:d6:e6:38:bf:1a

❌  Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for 9a:d6:e6:38:bf:1a

[nirs]

@adietish,

This is a minikube bug:

🔑  Your firewall is blocking bootpd which is required for this configuration. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

• These commands are not needed in macOS 15. We have code that should skip them but maybe it does not work in all cases
• The commands are not enough in older systems (macOS 14)
• Minikube should not try to run them and should not ask for your password - it is not secure to give minikube your password since it is typically not installed in a secure way and can be replaced by another another program running as your user.

The same issue existing with qemu and socket_vment, this is not a vfkit driver issue.

For the actual problem, we know that vmnet network on macOS is a little bit flaky specially on managed machines that use networking filters and dns that you cannot control.

Things that you can try when the minikube cluster does not get an IP address:

• Open System Settings > Network and turn the firewall off and on
• Terminate the socketfilterfw process
• Reboot

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale