higress icon indicating copy to clipboard operation
higress copied to clipboard

helm部署的higress 1.4.2版本升级到v2.0.0-rc.1时higress-controller一直报错Not ready to switch validation to fail-closed

Open ishells opened this issue 5 months ago • 2 comments

Ⅰ. Issue Description

Helm部署的higress 1.4.2版本,为了解决长链场景下批量添加/修改ingress引起的内存不释放问题,希望升级到higress 2.0版本,按照步骤升级到v2.0.0-rc.1后higress controller疯狂报错。

  • 操作步骤
  • 1、更新CRD资源
kubectl apply -f https://raw.githubusercontent.com/alibaba/higress/v2.0.0-rc.1/helm/core/crds/customresourcedefinitions.gen.yaml
  • 2、因为看到helm repo没有更新版本,就手动更改了higress controller 、 higress gateway的image tag为v2.0.0-rc.1,higress controller启动成功后便一直报错
2024-09-11T08:27:59.009727Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid rejected for the wrong reason: gateways.networking.istio.io is forbidden: User "system:serviceaccount:higress-test:higress-controller" cannot create resource "gateways" in API group "networking.istio.io" in the namespace "higress-test"
2024-09-11T08:27:59.022513Z	info	validationController	Successfully updated validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore,resourceVersion=1016868101)
2024-09-11T08:27:59.022544Z	error	controllers	error handling istio-validator-istio-system, retrying (retry count: 1): webhook is not ready, retry	controller=validation
2024-09-11T08:27:59.024060Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid rejected for the wrong reason: gateways.networking.istio.io is forbidden: User "system:serviceaccount:higress-test:higress-controller" cannot create resource "gateways" in API group "networking.istio.io" in the namespace "higress-test"
2024-09-11T08:27:59.034137Z	info	validationController	Successfully updated validatingwebhookconfiguration istiod-default-validator (failurePolicy=Ignore,resourceVersion=1016868102)
2024-09-11T08:27:59.034183Z	error	controllers	error handling istiod-default-validator, retrying (retry count: 1): webhook is not ready, retry	controller=validation
2024-09-11T08:27:59.035834Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid rejected for the wrong reason: gateways.networking.istio.io is forbidden: User "system:serviceaccount:higress-test:higress-controller" cannot create resource "gateways" in API group "networking.istio.io" in the namespace "higress-test"
2024-09-11T08:27:59.035857Z	info	validationController	validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore, resourceVersion=1016868101) is up-to-date. No change required.
2024-09-11T08:27:59.035877Z	error	controllers	error handling istio-validator-istio-system, retrying (retry count: 2): webhook is not ready, retry	controller=validation
2024-09-11T08:27:59.037174Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid rejected for the wrong reason: gateways.networking.istio.io is forbidden: User "system:serviceaccount:higress-test:higress-controller" cannot create resource "gateways" in API group "networking.istio.io" in the namespace "higress-test"
2024-09-11T08:27:59.037194Z	info	validationController	validatingwebhookconfiguration istiod-default-validator (failurePolicy=Ignore, resourceVersion=1016868102) is up-to-date. No change required.
2024-09-11T08:27:59.037208Z	error	controllers	error handling istiod-default-validator, retrying (retry count: 2): webhook is not ready, retry	controller=validation
2024-09-11T08:27:59.044542Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid rejected for the wrong reason: gateways.networking.istio.io is forbidden: User "system:serviceaccount:higress-test:higress-controller" cannot create resource "gateways" in API group "networking.istio.io" in the namespace "higress-test"
2024-09-11T08:27:59.051794Z	info	validationController	Successfully updated validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore,resourceVersion=1016868104)
2024-09-11T08:27:59.051820Z	error	controllers	error handling istio-validator-istio-system, retrying (retry count: 3): webhook is not ready, retry	controller=validation
2024-09-11T08:27:59.052880Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid rejected for the wrong reason: gateways.networking.istio.io is forbidden: User "system:serviceaccount:higress-test:higress-controller" cannot create resource "gateways" in API group "networking.istio.io" in the namespace "higress-test"
2024-09-11T08:27:59.052895Z	info	validationController	validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore, resourceVersion=1016868104) is up-to-date. No change required.
2024-09-11T08:27:59.052903Z	error	controllers	error handling istio-validator-istio-system, retrying (retry count: 4): webhook is not ready, retry	controller=validation

Ⅵ. Environment:

  • Higress version: 1.4.2
  • 集群内安装过istio,存在istio-validator
$ kubectl get ValidatingWebhookConfiguration | grep istio
istio-validator-istio-system                 1          519d
istiod-default-validator                     1          614d

ishells avatar Sep 11 '24 09:09 ishells