Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition in k8s `1.24.1`

[Divine1]

i'm trying to install ingress-nginx using helm on k8s cluster, but i receive below error Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition

i use k8s version 1.24.1 (screenshot attached)

i installed the k8s cluster using kubeadm The commands i used to install ingress-nginx using helm are present in the attached screenshot.

Please let me know how to fix this issue.

i didnot face this issue with k8s v1.22.9

[longwuyuan]

Do you have kind or minikube. Please try to install on minikube or kind. But please try 2 versions of K8S i.e v1.23.x and also v1.24.x on minikube or kind. There is #8091 because of which support for v1.24 is flaky.

Thanks, ; Long

On Tue, 14 Jun, 2022, 3:35 PM Kubernetes Prow Robot, < @.***> wrote:

@Divine1 https://github.com/Divine1: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here https://git.k8s.io/community/contributors/guide/pull-requests.md. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue: repository.

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/8690#issuecomment-1154981923, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWRJ2OXMK6ND7MOGKHTVPBKNLANCNFSM5YXDTIOQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Divine1]

@longwuyuan currently i destroyed that k8s cluster v1.24.1 and i have setup k8s cluster v1.22.9. v1.22.9 works fine for me as of now.

i already have a different project running on minikube k8s v1.22.9. i cannot create another one as of now. i will try to check with v1.23.x tomorrow ..

[longwuyuan]

/triage accepted

[erikgb]

I am also experiencing the same issue with the latest version of the ingress-nginx helm chart. Works fine with version 4.1.3, but with version 4.1.4 I am getting the same error message:

client.go:529: [debug] Watching for changes to Job ingress-nginx-admission-create with timeout of 5m0s
client.go:557: [debug] Add/Modify event for ingress-nginx-admission-create: ADDED
client.go:596: [debug] ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:557: [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:596: [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition
helm.go:84: [debug] failed pre-install: timed out waiting for the condition
INSTALLATION FAILED

I am running k3s version v1.23.6-k3s1.

[erikgb]

Also failing on k3s version v1.22.10-k3s1, so I think the Kubernetes version is unrelated. I just checked the job-pod locally, and the issue seems to be related to certificates, at least in my environment:

39s         Warning   Failed             pod/ingress-nginx-admission-create-95pm7   Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660": failed to resolve reference "registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660": failed to do request: Head "https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660": x509: certificate signed by unknown authority

Any clues?

[longwuyuan]

@erikgb This is very confusing. The project folks are aware of election-id being a issue. Tracked in #8091. And that implies lack of support on K8S v1.24 only and nothing else, purely based on the deprecated api.

Certificate or other issues are incidental and specific to the install and not a bug in the controller as far as K8S version below v1.24.x .

[erikgb]

@longwuyuan Sorry, my apologies. In my case, this was because of a MIM-problem after the migration to registry.k8s.io. Please ignore all my comments above.

[longwuyuan]

Thank you for updating. Glad not a concern.

[EwanRoycroft]

Same issue for me with k8s 1.24.2, helm 3.9.0.

$ helm install --create-namespace --namespace ingress-nginx ingress-nginx ingress-nginx/ingress-nginx --debug
install.go:178: [debug] Original chart version: ""
install.go:195: [debug] CHART PATH: /home/ubuntu/.cache/helm/repository/ingress-nginx-4.1.4.tgz

client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" ServiceAccount
client.go:339: [debug] serviceaccounts "ingress-nginx-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" ClusterRole
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" ClusterRoleBinding
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" Role
client.go:339: [debug] roles.rbac.authorization.k8s.io "ingress-nginx-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" RoleBinding
client.go:339: [debug] rolebindings.rbac.authorization.k8s.io "ingress-nginx-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission-create" Job
client.go:339: [debug] jobs.batch "ingress-nginx-admission-create" not found
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job ingress-nginx-admission-create with timeout of 5m0s
client.go:568: [debug] Add/Modify event for ingress-nginx-admission-create: ADDED
client.go:607: [debug] ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:607: [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition
helm.go:84: [debug] failed pre-install: timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
	helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:902
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:255
runtime.goexit
	runtime/asm_amd64.s:1581

[EwanRoycroft]

I reverted my cluster to k8s 1.22, as suggested by @Divine1 in the OP, but am getting the same issue:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.11", GitCommit:"5824e3251d294d324320db85bf63a53eb0767af2", GitTreeState:"clean", BuildDate:"2022-06-16T05:39:23Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.11", GitCommit:"5824e3251d294d324320db85bf63a53eb0767af2", GitTreeState:"clean", BuildDate:"2022-06-16T05:33:55Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

$ helm install --create-namespace --namespace ingress-nginx ingress-nginx ingress-nginx/ingress-nginx --debug
install.go:178: [debug] Original chart version: ""
install.go:195: [debug] CHART PATH: /home/ubuntu/.cache/helm/repository/ingress-nginx-4.1.4.tgz

client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" ServiceAccount
client.go:339: [debug] serviceaccounts "ingress-nginx-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" ClusterRole
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" ClusterRoleBinding
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" Role
client.go:339: [debug] roles.rbac.authorization.k8s.io "ingress-nginx-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission" RoleBinding
client.go:339: [debug] rolebindings.rbac.authorization.k8s.io "ingress-nginx-admission" not found
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "ingress-nginx-admission-create" Job
client.go:339: [debug] jobs.batch "ingress-nginx-admission-create" not found
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job ingress-nginx-admission-create with timeout of 5m0s
client.go:568: [debug] Add/Modify event for ingress-nginx-admission-create: ADDED
client.go:607: [debug] ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for ingress-nginx-admission-create: MODIFIED
client.go:607: [debug] ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition
helm.go:84: [debug] failed pre-install: timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
	helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:902
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:255
runtime.goexit
	runtime/asm_amd64.s:1581

[EwanRoycroft]

It seems my issue with k8s 1.22 was because I had forgotten to reinstall cillium when downgrading. I have now successfully installed on k8s 1.22.11.

[k8s-ci-robot]

@longwuyuan: The label(s) area/stabilisation cannot be applied, because the repository doesn't have them.

In response to this:

/area stabilisation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/area stabilization

[goafabric]

experiencing the same issue here when using kubernetes 1.24 inside docker .. helm chart fails

[longwuyuan]

The project released v1.3.0 of the controller with support for K8S v1.24.X . Please check and update

/remove-kind bug

[goafabric]

Sadly doesn't work for me:

• latest docker dekstop on Apple Silicon M1 with Kubernetes 1.24.1
• latest helm chart 4.20

Stil the same problem, helm charts hangs forever trying to install .. i will attach the logs though i can see nothing special in there.

Or is there something more we need to do ? Because the Changelog mentions something about special access rights. Without telling what to do ... so I expected the helm chart to contain these change _k8s_controller_ingress-nginx-controller-6bf7bc7f94-x42hg_ingress-nginx_dbdd0b9d-185e-4241-804c-52623df52b0d_0_logs.txt s

[tao12345666333]

let me take a look

[webcoderz]

same problem on:

kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

EDIT Solved it for me was a different problem not really applicable , system was airgapped and worked once I brought containers into the airgapped container registry

[appsailor]

I am not able to get it working in linode k8s :( Any progress?

The install/upgrade hangs and fails after 5 min timeout: ingress-nginx scoreblox-dev 2 2022-07-28 20:37:10.817710075 +0000 UTC failed ingress-nginx-4.1.3 1.2.1

related job stuck, not sure whats going on in there. ingress-nginx-admission-create 0/1 9m48s

• helm version

version.BuildInfo{Version:"v3.9.2", GitCommit:"1addefbfe665c350f4daf868a9adc5600cc064fd", GitTreeState:"clean", GoVersion:"go1.18.4"}

• Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8", GitCommit:"a12b886b1da059e0190c54d09c5eab5219dd7acf", GitTreeState:"clean", BuildDate:"2022-06-16T05:51:36Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}

[tao12345666333]

/assign

[goafabric]

is there any progress for this issue ?

@goafabric remind me

[longwuyuan]

Please try chart version 4.2.3 and update status.

[longwuyuan]

@Divine1 if your problem is solved by Chart version 4.2.3 or if no longer relevant, then kindly help and close the issue.

[goafabric]

@longwuyuan thank u, will do eventually

[Divine1]

@longwuyuan i have not checked 4.2.3 yet , but if you believe that it is fixed, then i believe you (ofcourse until i verify/confirm it).

Will reopen this ticket if any issue. Takecare. Have a nice day.

[longwuyuan]

@Divine1 there was a long awaited PR that merged recently. So we would love to get feedback/confirmation. Thank you very much for taking time to update.

[goafabric]

Feeback:

Problem persists with Helm Chart 4.2.3 On Latest Docker Desktop Kubernetes , Apple Silicon

Helm charts hangs with "waiting for condition". Ingress Controller gets into ready state 1/1 though .. but is unusable afterwards

Did anybody verify that the new version is working on Docker Desktop ?

[rosekizin]

@longwuyuan

FeedBack:

Problem still persists in Helm Chart 4.2.3

Warning  Failed     11m                  kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47": failed to resolve reference "registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47": failed to do request: Head "https://registry.k8s.io/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47": dial tcp: lookup registry.k8s.io on [::1]:53: read udp [::1]:38535->[::1]:53: read: connection refused

The nginx controller service is not installed and controller admission looks like

It seems the DNS server is blocking the connection and preventing my Kubernetes download the image.

[goafabric]

@longwuyuan i dont know if you see the comments, because the issue is already closed but unfortunately the issue still persists for me inside docker desktip kubernetes

"Helm charts hangs with "waiting for condition". Ingress Controller gets into ready state 1/1 though .. but is unusable afterwards"

Did anybody verify that its working now within docker desktop ?

@Divine1 Could you verify if this is now working for you or not ?

[longwuyuan]

@goafabric, please find people to discuss this with on the kubernetes slack in the ingress-nginx-users channel.