crc-org
crc start fails on linux with bundle 4.19.13: Temporary error: pull secret not updated to disk (x206)
General information
CRC cluster fails to start completely with the error "Failed to update pull secret on the disk: Temporary error: pull secret not updated to disk (x206)".
The probable cause appears to be an expired kube-scheduler client certificate.
This is only happening with bundle 4.19.13, 4.19.8 is not having this issue. It reproduces with both 2.55.0 and 2.55.1
Operating System
Linux
Hypervisor
KVM
Did you run crc setup before crc start?
yes
Running on
Baremetal-Server
Steps to reproduce
- Run crc start
- Wait for the cluster initialization process
- Observe the failure after ~10 minutes with the pull secret update error
CRC version
CRC version: 2.55.1+6252bc
OpenShift version: 4.19.13
MicroShift version: 4.19.7
CRC status
ERRO crc does not seem to be setup correctly, have you run 'crc setup'?
CRC config
- bundle : https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/bundles/openshift/4.19.13/crc_libvirt_4.19.13_arm64.crcbundle
- consent-telemetry : no
- cpus : 54
- disk-size : 51
- memory : 109254
- network-mode : system
- pull-secret-file : pull-secret.txt
- skip-check-daemon-systemd-sockets : true
- skip-check-daemon-systemd-unit : true
- skip-check-ram : true
- skip-check-systemd-networkd-running : true
- skip-check-systemd-resolved-running : true
- skip-check-virt-enabled : true
Host Operating System
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
Expected behavior
CRC cluster should start successfully and complete all initialization steps including updating the pull secret on disk.
Actual behavior
CRC VM starts successfully and the cluster begins initialization, but fails during the final stages with "Failed to update pull secret on the disk: Temporary error: pull secret not updated to disk (x206)" after waiting for cluster stabilization.
CRC Logs
+ crc start
INFO Using bundle path https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/bundles/openshift/4.19.13/crc_libvirt_4.19.13_arm64.crcbundle
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking if running on a supported CPU architecture
WARN CRC is not officially supported on ARM64 CPUs for Linux.
INFO Checking if crc executable symlink exists
INFO Checking minimum RAM requirements
WARN Skipping above check...
INFO Check if Podman binary exists in: /home/runner/.crc/bin/oc
INFO Checking if Virtualization is enabled
WARN Skipping above check...
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
WARN Skipping above check...
INFO Checking if AppArmor is configured
INFO Checking if systemd-networkd is running
WARN Skipping above check...
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if dnsmasq configurations file exist for NetworkManager
INFO Checking if the systemd-resolved service is running
WARN Skipping above check...
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists
INFO Checking if libvirt 'crc' network is available
INFO Checking if libvirt 'crc' network is active
INFO Loading bundle: crc_libvirt_4.19.13_arm64...
INFO Creating CRC VM for OpenShift 4.19.13...
INFO Generating new SSH key pair...
INFO Generating new password for the kubeadmin user
INFO Starting CRC VM for openshift 4.19.13...
INFO CRC instance is running with IP 192.168.130.11
INFO CRC VM is running
INFO Updating authorized keys...
INFO Resizing /dev/vda4 filesystem
INFO Configuring shared directories
INFO Check internal and public DNS query...
INFO Check DNS query from host...
WARN Wildcard DNS resolution for apps-crc.testing does not appear to be working
INFO Verifying validity of the kubelet certificates...
INFO Starting kubelet service
INFO Waiting for kube-apiserver availability... [takes around 2min]
INFO Adding user's pull secret to the cluster...
INFO Updating SSH key to machine config resource...
INFO Overriding password for developer user
INFO Changing the password for the users
INFO Updating cluster ID...
INFO Updating root CA cert to admin-kubeconfig-client-ca configmap...
INFO Starting openshift instance... [waiting for the cluster to stabilize]
WARN Cluster is not ready: cluster operators are still not stable after 10m0.101770067s
INFO Waiting until the user's pull secret is written to the instance disk...
ERROR Failed to update pull secret on the disk: Temporary error: pull secret not updated to disk (x206)
Additional context
The probable cause appears to be an expired kube-scheduler client certificate. Certificate expiry check shows:
oc -n openshift-kube-scheduler get secret kube-scheduler-client-cert-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate -subject -issuer
notAfter=Oct 24 08:31:41 2025 GMT
subject=CN = system:kube-scheduler
issuer=OU = openshift, CN = kube-control-plane-signer
The certificate expired on Oct 24, 2025, which may be preventing the cluster from stabilizing and completing the pull secret update process.
Workaround
Installed the version 4.19.8 instead, it worked fine (https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/2.54.0).
Same problem here. Installed the version 4.19.8 instead, it worked fine (https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/2.54.0).
INFO Updating root CA cert to admin-kubeconfig-client-ca configmap...
INFO Starting openshift instance... [waiting for the cluster to stabilize]
WARN Cluster is not ready: cluster operators are still not stable after 10m0.101770067s
^^ This looks like cluster is not even stable after default timeout. It may be because of slow system or nested virt environment. Is this run on laptop or bare metal machine or happening some CI environment?
@praveenkumar
I posted the probable cause in the Additional context section, and you can see the machine size in the CRC config section. The issue can be easily reproduced, even on a laptop.
Looks like it is due to bundle cert expiration which is not working https://github.com/crc-org/snc/issues/1181 , this is tracker issue for us. We will update more details and create new bundle with long cert period.
Using 4.19 latest same bug, will not work on: Windows, RHEL10, Fedora and Ubuntu.
Will the same bundle be updated? Or will this be fixed in next release?
@codersyacht it is going to fix in next release. there is no bundle update planned,
I'm not sure what your release cadence is. Should we be expecting a fix for this soon, a couple of weeks, or longer? I'm unsure how to go about testing a fix without a release.
I'm not sure what your release cadence is. Should we be expecting a fix for this soon, a couple of weeks, or longer? I'm unsure how to go about testing a fix without a release
@LightGuard we are trying to do the release this week if possible. meanwhile you can use workaround mentioned in the issue
Installed the version 4.19.8 instead, it worked fine (https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/2.54.0).
Having same issue with 4.19.13 on Macbook. Using workaround 4.19.8 works ok
Have the same problem, both on fedora 42 and 43. looking into the logs of the VM, it says:
Nov 07 14:39:59 crc kubenswrapper[4797]: E1107 14:39:59.552264 4797 kubelet.go:3017] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
I can confirm that this same issue exists on Windows with Hyper-V as well. 4.19.8 (crc 2.54.0) works fine though.
Is this going to be fixed in 4.19 or only in 4.20? We currently use 4.19, and the workaround version 4.19.8 is affected by this issue (fixed in 4.19.10): https://access.redhat.com/solutions/7129484
@jmunozro This should be only fixed in 4.20 as of now. if you really need 4.19.x bundle then you can create using https://github.com/crc-org/snc/tree/release-4.19 one.
We just did a new 2.56.0 release which should have fix for it because now we have long term certs. Please do try it and let us know if something still broken.
The installer from https://console.redhat.com/openshift/create/local does not seem to be accessible. Browser waits for a response before returning
An error occurred while processing your request.
Reference #221.dcd5ce17.1763416584.2d538459
https://errors.edgesuite.net/221.dcd5ce17.1763416584.2d538459
curl also fails to resolve.
The installer from https://console.redhat.com/openshift/create/local does not seem to be accessible. Browser waits for a response before returning
An error occurred while processing your request. Reference #221.dcd5ce17.1763416584.2d538459 https://errors.edgesuite.net/221.dcd5ce17.1763416584.2d538459curl also fails to resolve.
Seems problem with developers.redhat.com.
First attempt timed out in 4 minutes for just listing the directory :thinking:
$ time curl -I https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/latest/
HTTP/2 504
mime-version: 1.0
content-type: text/html
content-length: 278
expires: Tue, 18 Nov 2025 04:07:57 GMT
cache-control: max-age=0, no-cache, no-store
pragma: no-cache
date: Tue, 18 Nov 2025 04:07:57 GMT
x-rh-edge-request-id: 210e92
x-rh-edge-reference-id: 0.a05532b8.1763438637.210e92
x-rh-edge-cache-status: Error from child
strict-transport-security: max-age=86400 ; includeSubDomains
real 4m0.218s
user 0m0.076s
sys 0m0.012s
$ time curl -I https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/latest/
HTTP/2 200
content-type: text/html;charset=UTF-8
expires: Tue, 18 Nov 2025 04:08:36 GMT
cache-control: max-age=0, no-cache, no-store
pragma: no-cache
date: Tue, 18 Nov 2025 04:08:36 GMT
x-rh-edge-request-id: f45a393
x-rh-edge-reference-id: 0.9f5532b8.1763438916.f45a393
x-rh-edge-cache-status: Miss from child, Hit from parent
strict-transport-security: max-age=86400 ; includeSubDomains
real 0m0.415s
user 0m0.023s
sys 0m0.022s
There is an outage of developers.redhat.com https://status.redhat.com/ details about it, team is working on it to resolve it.
We have added the binary to GH release page also because of the outage, so please grab from there and try. you might need to disable the version update check otherwise it will take time to fetch next updated version from developers.redhat.com and timeout.
crc config set disable-update-check true
I observed the openshift 4.20.1 deployed by latest crc 2.56.0 could not show the operators in its new "Software Catalog"
The "Software Catalog" so far only showing "Builder Images", Devfiles, Helm Charts and Templates.
Operators are missing:
A colleague helped to check with normal installation on 4.20 (non-crc), and he did not seem seeing this issue on his side.
So, I assume this is issue on crc openshift local?
Are some of you seeing same issue on it above?
@felixmarch yes internally we also got this issue reported, looks like on cli side everything is working but on console side operators are missing from catalog, we are looking into this.
@jmunozro This should be only fixed in 4.20 as of now. if you really need 4.19.x bundle then you can create using https://github.com/crc-org/snc/tree/release-4.19 one.
Any way I can build this one in a similar way the official bundles are built? I got this error with a bundle built with your instructions:
level=fatal msg="Preset openshift is used but bundle is provided for okd preset"
Edit: I know I can change the profile with crc config set preset okd, but OKD is not passing our tests. It is inconvenient that the last published bundle (4.19.13) is broken, as we usually tests our software in the N previous versions, not only the latest one.
@jmunozro in that case you can use an older version of 4.19 bundle 4.19.8 which works as expected https://mirror.openshift.com/pub/openshift-v4/clients/crc/bundles/openshift/4.19.8/ ?
That version has a regression that is breaking some of our tests: https://access.redhat.com/solutions/7129484