crc
crc copied to clipboard
crc-org
[Error] crc start : Error starting machine: Error in driver during machine start: Unable to detemine VM's IP address, did it fail to boot?
General information
OS: Windows
Hypervisor: Vmware Workstation 16 Pro
Did you run crc setup before starting it (Yes) But it fails
Running CRC on: Dell Desktop
CRC version
CRC version: 2.15.0+cc05160
OpenShift version: 4.12.15
Podman version: 4.3.1
CRC status
CRC VM: Running
OpenShift: Unreachable (v4.12.5)
RAM Usage: 511.5MB of 9.399GB
Disk Usage: 12.65GB of 32.74GB (Inside the CRC VM)
Cache Usage: 16.06GB
Cache Directory: /home/student/.crc/cache
CRC config
consent-telemetry : no
Host Operating System
NAME="Red Hat Enterprise Linux"
VERSION="9.1 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.1 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"
Red Hat Enterprise Linux release 9.1 (Plow)
Red Hat Enterprise Linux release 9.1 (Plow)
processor : 0-7
vendor_id : GenuineIntel
cpu family : 6
model : 167
model name : 11th Gen Intel(R) Core(TM) i9-11900KF @ 3.50GHz
stepping : 1
microcode : 0x57
cpu MHz : 3503.999
cache size : 16384 KB
physical id : 1
siblings : 4
core id : 3
cpu cores : 4
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 27
wp : yes
Steps to reproduce
# crc delete -f
Deleted the instance
# crc cleanup
INFO Removing vsock configuration
INFO Removing 'crc' network from libvirt
INFO Removing /etc/NetworkManager/dnsmasq.d/crc.conf file
INFO Using root access: Removing NetworkManager configuration file in /etc/NetworkManager/dnsmasq.d/crc.conf
INFO Using root access: Executing systemctl daemon-reload command
INFO Using root access: Executing systemctl reload NetworkManager
INFO Removing /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf file
INFO Using root access: Removing NetworkManager configuration file in /etc/NetworkManager/conf.d/crc-nm-dnsmasq.con
INFO Using root access: Executing systemctl daemon-reload command
INFO Using root access: Executing systemctl reload NetworkManager
INFO Removing crc daemon systemd socket units
INFO Removing crc daemon systemd service
INFO Removing crc's virtual machine
INFO Removing crc libvirt storage pool
INFO Removing hosts file records added by CRC
INFO Removing pull secret from the keyring
INFO Removing older logs
INFO Removing CRC Machine Instance directory
INFO Removing crc executable symlink
Cleanup finished
# crc setup
INFO Using bundle path /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Creating symlink for crc executable
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd service
INFO Setting up crc daemon systemd service
INFO Checking crc daemon systemd socket units
INFO Setting up crc daemon systemd socket units
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
INFO Writing Network Manager config for crc
INFO Using root access: Writing NetworkManager configuration to /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf
INFO Using root access: Changing permissions for /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf to 644
INFO Using root access: Executing systemctl daemon-reload command
INFO Using root access: Executing systemctl reload NetworkManager
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
INFO Writing dnsmasq config for crc
INFO Using root access: Writing NetworkManager configuration to /etc/NetworkManager/dnsmasq.d/crc.conf
INFO Using root access: Changing permissions for /etc/NetworkManager/dnsmasq.d/crc.conf to 644
INFO Using root access: Executing systemctl daemon-reload command
INFO Using root access: Executing systemctl reload NetworkManager
INFO Checking if libvirt 'crc' network is available
INFO Setting up libvirt 'crc' network
INFO Checking if libvirt 'crc' network is active
INFO Starting libvirt 'crc' network
INFO Checking if CRC bundle is extracted in '$HOME/.crc'
INFO Checking if /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists
Your system is correctly setup for using CRC. Use 'crc start' to start the instance
# crc start
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
INFO Checking if libvirt 'crc' network is available
INFO Checking if libvirt 'crc' network is active
INFO Loading bundle: crc_libvirt_4.12.5_amd64...
CRC requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://console.redhat.com/openshift/create/local.
? Please enter the pull secret *************************************************************************************
INFO Creating CRC VM for openshift 4.12.5...
INFO Generating new SSH key pair...
INFO Generating new password for the kubeadmin user
INFO Starting CRC VM for openshift 4.12.5...
**** wait for good old 20 minutes. Then this ***
Error starting machine: Error in driver during machine start: Unable to determine VM's IP address, did it fail to boot?
View logs
# crc setup --log-level debug
DEBU CRC version: 2.15.0+cc05160
DEBU OpenShift version: 4.12.5
DEBU Podman version: 4.3.1
DEBU Running 'crc setup'
INFO Using bundle path /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle
DEBU Checking if systemd-resolved.service is running
DEBU Running 'systemctl status systemd-resolved.service'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit systemd-resolved.service could not be found.
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
DEBU Running '/home/student/.crc/bin/crc-admin-helper-linux --version'
DEBU Found crc-admin-helper-linux version 0.0.12
DEBU crc-admin-helper executable already cached
INFO Checking if running on a supported CPU architecture
DEBU GOARCH is amd64 GOOS is linux
INFO Checking minimum RAM requirements
DEBU Total memory of system is 12240977920 bytes
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo
DEBU CPU virtualization flags are good
INFO Checking if KVM is enabled
DEBU Checking if /dev/kvm exists
DEBU /dev/kvm was found
INFO Checking if libvirt is installed
DEBU Checking if 'virsh' is available
DEBU 'virsh' was found in /usr/bin/virsh
DEBU Checking 'virsh capabilities' for libvirtd/qemu availability
DEBU Running 'virsh --readonly --connect qemu:///system capabilities'
DEBU Found x86_64 hypervisor with 'hvm' capabilities
INFO Checking if user is part of libvirt group
DEBU Checking if current user is part of the libvirt group
DEBU Current user is already in the libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
DEBU Checking if libvirtd service is running
DEBU Running 'systemctl status virtqemud.socket'
DEBU Command failed: exit status 3
DEBU stdout: * virtqemud.socket - Libvirt qemu local socket
Loaded: loaded (/usr/lib/systemd/system/virtqemud.socket; enabled; vendor preset: disabled)
Active: inactive (dead)
Triggers: * virtqemud.service
Listen: /run/libvirt/virtqemud-sock (Stream)
DEBU stderr:
DEBU virtqemud.socket is neither running nor listening
DEBU Running 'systemctl status libvirtd.socket'
DEBU libvirtd.socket is running
INFO Checking if a supported libvirt version is installed
DEBU Checking if libvirt version is >=3.4.0
DEBU Running 'virsh -v'
INFO Checking if crc-driver-libvirt is installed
DEBU Checking if crc-driver-libvirt is installed
DEBU Running '/home/student/.crc/bin/crc-driver-libvirt version'
DEBU Found crc-driver-libvirt version 0.13.5
DEBU crc-driver-libvirt is already installed
INFO Checking crc daemon systemd service
DEBU Checking crc daemon systemd service
DEBU Checking if crc-daemon.service is running
DEBU Running 'systemctl --user status crc-daemon.service'
DEBU Command failed: exit status 3
DEBU stdout: * crc-daemon.service - CRC daemon
Loaded: loaded (/home/student/.config/systemd/user/crc-daemon.service; static)
Active: inactive (dead)
TriggeredBy: * crc-vsock.socket
* crc-http.socket
Mar 26 09:56:49 crc2.example.com systemd[2233]: Starting CRC daemon...
Mar 26 09:56:49 crc2.example.com crc[8841]: level=info msg="using socket provided by crc-vsock.socket"
Mar 26 09:56:49 crc2.example.com crc[8841]: level=info msg="using socket provided by crc-http.socket"
Mar 26 09:56:49 crc2.example.com systemd[2233]: Started CRC daemon.
Mar 26 09:56:50 crc2.example.com crc[8841]: @ - - [26/Mar/2023:09:56:49 +0800] "GET /api/status HTTP/1.1" 200 189
Mar 26 09:57:10 crc2.example.com crc[8841]: @ - - [26/Mar/2023:09:57:09 +0800] "GET /api/status HTTP/1.1" 200 189
Mar 26 10:07:42 crc2.example.com systemd[2233]: Stopping CRC daemon...
Mar 26 10:07:42 crc2.example.com systemd[2233]: Stopped CRC daemon.
DEBU stderr:
DEBU crc-daemon.service is neither running nor listening
DEBU Checking if crc-daemon.service has the expected content
INFO Checking crc daemon systemd socket units
DEBU Checking crc daemon systemd socket units
DEBU Checking if crc-http.socket is running
DEBU Running 'systemctl --user status crc-http.socket'
DEBU crc-http.socket is listening
DEBU Checking if crc-http.socket has the expected content
DEBU Checking if crc-vsock.socket is running
DEBU Running 'systemctl --user status crc-vsock.socket'
DEBU crc-vsock.socket is listening
DEBU Checking if crc-vsock.socket has the expected content
INFO Checking if systemd-networkd is running
DEBU Checking if systemd-networkd.service is running
DEBU Running 'systemctl status systemd-networkd.service'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit systemd-networkd.service could not be found.
DEBU systemd-networkd.service is not running
INFO Checking if NetworkManager is installed
DEBU Checking if 'nmcli' is available
DEBU 'nmcli' was found in /usr/bin/nmcli
INFO Checking if NetworkManager service is running
DEBU Checking if NetworkManager.service is running
DEBU Running 'systemctl status NetworkManager.service'
DEBU NetworkManager.service is already running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
DEBU Checking NetworkManager configuration
DEBU NetworkManager configuration is good
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
DEBU Checking dnsmasq configuration
DEBU dnsmasq configuration is good
INFO Checking if libvirt 'crc' network is available
DEBU Checking if libvirt 'crc' network exists
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU Checking if libvirt 'crc' definition is up to date
DEBU Running 'virsh --connect qemu:///system net-dumpxml --inactive crc'
DEBU libvirt 'crc' network has the expected value
INFO Checking if libvirt 'crc' network is active
DEBU Checking if libvirt 'crc' network is active
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU libvirt 'crc' network is already active
INFO Checking if CRC bundle is extracted in '$HOME/.crc'
INFO Checking if /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists
DEBU /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists
Your system is correctly setup for using CRC. Use 'crc start' to start the instance
This looks like a nested virtualization setup, @jasonwcc any reason why don't you start crc directly on the windows using hyperV instead nested virtualization setup? We have close to no testing around nested virtualization side.
Actually, prior to this v4.12.5, the older version CRC all working wonderful over my nested virtualization using Vmware workstation. only this version pose this issue. My company dictate that I am not supposed to use my local laptop windows to setup as hypervisor, hence the use of vmware workstaiton or virtual box as mean of testing and training.
Thank you and Warmest Regards
On Monday, March 27, 2023 at 01:29:53 PM GMT+8, Praveen Kumar ***@***.***> wrote:
This looks like a nested virtualization setup, @jasonwcc any reason why don't you start crc directly on the windows using hyperV instead nested virtualization setup? We have close to no testing around nested virtualization side.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
@jasonwcc what was the last crc version you tried and worked as expected?
Hi following version: 4.11.184.10.184.10.224.10.124.9.15 practically since 4.9.x to 4.11.x4.12.5 is the first version i cant use Thank you and Warmest Regards
On Wednesday, March 29, 2023 at 03:51:18 PM GMT+8, Praveen Kumar ***@***.***> wrote:
@jasonwcc what was the last crc version you tried and worked as expected?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
@jasonwcc there is lot of things changes from each openshift release on OS level also (like kernel package updates) we are not sure what changed and not working for you. We also don't have any CI in place to do nested virtualization test for these kind of setup and mentioned in the doc.
Error starting machine: Error in driver during machine start: Unable to determine VM's IP address, did it fail to boot?
Most likely a VMware related issue. Without further info it is hard to determine the cause. Please try to add more info, such as console output. You could try to disable any VPN you run and IPv6.
Unsupported.
I face exact the same issue with crc 2.13.1+3b466f8 on RHEL 9.1. Similar issue found at https://github.com/crc-org/crc/issues/2397 which is not really fixed. If I try multiple times and reboot my VM, it may succeed occasionally. But this is still an issue to me.
[myuser@myhost crc-linux-2.13.1-amd64]$ ./crc start -m 24000 -c 8 -d 100
WARN A new version (2.20.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.20.0/crc-linux-amd64.tar.xz
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
INFO Checking if libvirt 'crc' network is available
INFO Checking if libvirt 'crc' network is active
INFO Loading bundle: crc_libvirt_4.12.0_amd64...
CRC requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://console.redhat.com/openshift/create/local.
? Please enter the pull secret ****************************************************************************************************************************************************************
INFO Creating CRC VM for openshift 4.12.0...
INFO Generating new SSH key pair...
INFO Generating new password for the kubeadmin user
INFO Starting CRC VM for openshift 4.12.0...
Error starting machine: Error in driver during machine start: Unable to determine VM's IP address, did it fail to boot?
At this point 4.12.0 is quite old, any chance you could try a newer version? Either the latest release with a 4.12.x bundle, or the latest release which ships with OpenShift 4.13? And is this with nested virtualization?
Hey, same issue here but with a different problem approach. What I'm running :
- A latest fedora 40VM vm in a ESXII7 (the vm has nested virtualziaicon applied in ESXII config)
- Crc version by the moment: 2.39.0+0849cb
- Openshift version: 4.16.0
- libvirtd version: 10.1.0
- dnsmasq version 2.90
- hostname of the vm: localhost
I have found the problem is related with the service libvirtd as is shown in the following exmaple:
- I run crc start and I get this:
crc start --log-level debug
DEBU CRC version: 2.39.0+0849cb
DEBU OpenShift version: 4.16.0
DEBU Running 'crc start'
DEBU Total memory of system is 15705567232 bytes
DEBU No new version available. The latest version is 2.39.0
DEBU Checking file: /home/user/.crc/machines/crc/.crc-exist
DEBU Found binary path at /home/user/.crc/bin/crc-driver-libvirt-amd64
DEBU Launching plugin server for driver libvirt
DEBU Plugin server listening at address 127.0.0.1:40199
DEBU () Calling .GetVersion
DEBU Using API Version 1
DEBU () Calling .SetConfigRaw
DEBU () Calling .GetMachineName
DEBU (crc) Calling .GetBundleName
DEBU (crc) Calling .GetState
DEBU (crc) DBG | time="2024-07-25T19:12:15+02:00" level=debug msg="Getting current state..."
DEBU (crc) DBG | time="2024-07-25T19:12:15+02:00" level=debug msg="Fetching VM..."
DEBU Making call to close driver server
DEBU (crc) Calling .Close
DEBU (crc) DBG | time="2024-07-25T19:12:15+02:00" level=debug msg="Closing plugin on server side"
DEBU Failed to make call to close driver server: unexpected EOF
DEBU Making call to close connection to plugin binary
DEBU Checking file: /home/user/.crc/machines/crc/.crc-exist
INFO Loading bundle: crc_libvirt_4.16.0_amd64...
DEBU Checking file: /home/user/.crc/machines/crc/.crc-exist
DEBU Found binary path at /home/user/.crc/bin/crc-driver-libvirt-amd64
DEBU Launching plugin server for driver libvirt
DEBU Plugin server listening at address 127.0.0.1:32901
DEBU () Calling .GetVersion
DEBU Using API Version 1
DEBU () Calling .SetConfigRaw
DEBU () Calling .GetMachineName
DEBU (crc) Calling .GetBundleName
DEBU (crc) Calling .GetState
DEBU (crc) DBG | time="2024-07-25T19:12:16+02:00" level=debug msg="Getting current state..."
DEBU (crc) DBG | time="2024-07-25T19:12:16+02:00" level=debug msg="Fetching VM..."
INFO A CRC VM for OpenShift 4.16.0 is already running
DEBU Making call to close driver server
DEBU (crc) Calling .Close
DEBU (crc) DBG | time="2024-07-25T19:12:16+02:00" level=debug msg="Closing plugin on server side"
DEBU Failed to make call to close driver server: unexpected EOF
DEBU Making call to close connection to plugin binary
Started the OpenShift cluster.
INFO Loading bundle: crc_libvirt_4.16.0_amd64...
INFO A CRC VM for OpenShift 4.16.0 is already running
Started the OpenShift cluster.
The server is accessible via web console at:
https://console-openshift-console.apps-crc.testing
Log in as administrator:
Username: kubeadmin
Password: 123456789
Log in as user:
Username: developer
Password: developer
Use the 'oc' command line interface:
$ eval $(crc oc-env)
$ oc login -u developer https://api.crc.testing:6443
- In other terminal check for libvirtd service logs (localhost is the name of the fedora40 machine)
sudo journalctl -u libvirtd -f
Jul 25 19:10:44 localhost systemd[1]: Starting libvirtd.service - libvirt legacy monolithic daemon...
Jul 25 19:10:45 localhost systemd[1]: Started libvirtd.service - libvirt legacy monolithic daemon.
Jul 25 19:11:03 localhost libvirtd[1781]: libvirt version: 10.1.0, package: 3.fc40 (Fedora Project, 2024-07-09-11:57:17, )
Jul 25 19:11:03 localhost libvirtd[1781]: hostname: localhost
Jul 25 19:11:03 localhost libvirtd[1781]: End of file while reading data: Input/output error
Jul 25 19:11:03 localhost libvirtd[1781]: End of file while reading data: Input/output error
- There is a relation between everytime I run crc start with a new errror of: Jul 25 18:53:20 localhost libvirtd[4632]: End of file while reading data: Input/output error
I have trid to restart the libvirtd service and the dnsmasq service, but still not workking.
@valkiriaaquatica nested virtualization is something not tested by us also as per logs I can see that cluster is already running so I am not sure what is the issue you are facing can you delete the cluster and provide the output of following commands (may be in a gist)
$ crc delete
$ crc setup --log-level debug
$ crc start --log-level debug
$ sudo cat /var/log/libvirt/qemu/crc.log