ingress-nginx icon indicating copy to clipboard operation
ingress-nginx copied to clipboard

Multiple error in while election process

Open Asgoret opened this issue 1 year ago • 6 comments

What happened:

NGINX pods can't create and update ingress-controller-leader configmap. Errors:

  • Creating: error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:ingress-controller:nginx-ingress-ingress-nginx" cannot create resource "configmaps" in API group "" in the namespace "ingress-controller"
  • Update: Failed to update lock: configmaps "ingress-controller-leader" is forbidden: User "system:serviceaccount:ingress-controller:nginx-ingress-ingress-nginx" cannot update resource "configmaps" in API group "" in the namespace "ingress-controller"

Solution: Add to role ingress-nginx:

- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - create
  - update

What you expected to happen:

All works fine)

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): NGINX Ingress controller Release: v1.3.0 Build: 2b7b74854d90ad9b4b96a5011b9e8b67d20bfb8f Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.19.10

Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:40Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.24) and server (1.27) exceeds the supported minor version skew of +/-1

Helm version: 4.8.2

Asgoret avatar Oct 18 '23 13:10 Asgoret

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Oct 18 '23 13:10 k8s-ci-robot

/remove-kind bug

  • Please install as per documentation https://kubernetes.github.io/ingress-nginx/deploy/ and update
  • When you update, please answer the questions that are asked in the new issue template as that info is required to take any action

longwuyuan avatar Oct 19 '23 10:10 longwuyuan

@longwuyuan I installed ingress ad documentations said. If u look closely on role u see, that it don't have necessary verbs. https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml image

Asgoret avatar Oct 19 '23 12:10 Asgoret

In the latest release the leader election happens based on lease and not on configmap. If you are creating your own roles then you need to handle appropriately based on the roles provided in that branch

bmv126 avatar Oct 22 '23 06:10 bmv126

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

github-actions[bot] avatar Nov 22 '23 01:11 github-actions[bot]

I have exactly the same issue with fresh installation of ingress-nginx from helm chart.

E0404 12:16:22.587172 7 leaderelection.go:334] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:nginx-internal-v2:ingress-nginx-internal-v2" cannot create resource "configmaps" in API group "" in the namespace "nginx-internal-v2" E0404 12:16:36.819387 7 leaderelection.go:334] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:nginx-internal-v2:ingress-nginx-internal-v2" cannot create resource "configmaps" in API group "" in the namespace "nginx-internal-v2" E0404 12:16:49.169609 7 leaderelection.go:334] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:nginx-internal-v2:ingress-nginx-internal-v2" cannot create resource "configmaps" in API group "" in the namespace "nginx-internal-v2" E0404 12:16:58.691519 7 leaderelection.go:334] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:nginx-internal-v2:ingress-nginx-internal-v2" cannot create resource "configmaps" in API group "" in the namespace "nginx-internal-v2"

after applying the solution suggested by @Asgoret ingress-nginx is able to get election.

I0404 12:17:10.082047 7 leaderelection.go:258] successfully acquired lease nginx-internal-v2/ingress-controller-leader I0404 12:17:10.082139 7 status.go:84] "New leader elected" identity="ingress-nginx-internal-v2-controller-5bd4f9d4bc-cgfdm"

fresh installation in new namespace, chart version = ingress-nginx-4.10.0 helm version = version.BuildInfo{Version:"v3.14.3", GitCommit:"f03cc04caaa8f6d7c3e67cf918929150cf6f3f12", GitTreeState:"clean", GoVersion:"go1.22.1"}

kubectl version: Client Version: v1.29.0 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.6-eks-508b6b3

pmartyniuk avatar Apr 04 '24 12:04 pmartyniuk

I can not reproduce and leasder election was changed to use lease-api as it was posted earlier.

/close

longwuyuan avatar Sep 15 '24 14:09 longwuyuan

@longwuyuan: Closing this issue.

In response to this:

I can not reproduce and leasder election was changed to use lease-api as it was posted earlier.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot avatar Sep 15 '24 14:09 k8s-ci-robot