ingress-nginx
ingress-nginx copied to clipboard
kubernetes
validatingwebhook doesn't work as expected when enableAnnotationValidations is set to true
What happened:
Create the below bad-ingress with proxy-next-upstream set to 300. It creates an ingress object even though the proxy-next-upstream value is in-correct.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: bad-ingress
namespace: default
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-next-upstream: "300"
spec:
rules:
- host: bad-ingress.k8s.mydomain
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: bad-ingress-service
port:
number: 8888
This issue occurs only when enableAnnotationValidations is set to true.
What you expected to happen:
Ideally, the validating webhook should reject creating this ingress object. Looks like the additional annotation validation that's introduced in v1.9.0 is not validating the errors as expected.
kubectl apply -f bad-ingress.yaml
Error from server (BadRequest): error when creating "bad-ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
-------------------------------------------------------------------------------
Error: exit status 1
2023/11/23 08:15:04 [emerg] 297#297: invalid value "300" in /tmp/nginx/nginx-cfg910596062:568
nginx: [emerg] invalid value "300" in /tmp/nginx/nginx-cfg910596062:568
nginx: configuration file /tmp/nginx/nginx-cfg910596062 test failed
-------------------------------------------------------------------------------
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
NGINX Ingress controller Release: v1.9.4 Build: 846d251814a09d8a5d8d28e2e604bfc7749bcb49 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.21.6
Kubernetes version (use kubectl version):
kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:49Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Environment: On-Prem installed via kubeadm
- How was the ingress-nginx-controller installed:
helm list -n ingress-nginx
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
test-ingress ingress-nginx 6 2023-11-23 01:47:48.101692406 +0530 IST deployed ingress-nginx-4.8.3 1.9.4
This issue is currently awaiting triage.
If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
I think the webhookvalidation and the annotation value string validation are different code-paths and I guess non test is written to check for a value like 300. Probable that no upstream can be named as 300
Hello @srikiz Is that issue resolved ? thanks
Is a resolution expected in 1.9.5? I didn't my see a linked PR so we haven't tested 1.9.5 yet.
Hello everyone
look like i facing off with enableAnnotationValidations too.
please take a look
depend on CVE-2021-25742 ( This issue has been rated High )
but when I edited annotations-risk-level: High and set --enable-annotation-validation=true , look like this wrong and cannot create ingress object. Im not sure CVE-2021-25742 was rated correctly.
Here are details
- Configmap Ingress Nginx Controller
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
data:
allow-snippet-annotations: "true"
annotations-risk-level: High
- Deployment Ingress Nginx Controller
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: platform
app.kubernetes.io/component: controller
replicas: 5
revisionHistoryLimit: 10
minReadySeconds: 0
template:
spec:
containers:
- name: controller
image: registry.k8s.io-ingress-nginx-controller:v1.9.5
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- **--enable-annotation-validation=true**
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
- --enable-ssl-passthrough=true
- Deploy app with Ingress object which got issue. Exactly app is deploy Kiali inside namespace Istio
ingress:
class_name: "nginx"
enabled: true
# default: override_yaml is undefined
override_yaml:
metadata:
annotations:
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header "X-B3-Sampled" "1";
spec:
rules:
- host: kiali.example.com
http:
paths:
- backend:
service:
name: kiali
port:
number: 20001
path: /
pathType: Prefix
tls:
- hosts:
- kiali.example.com
secretName: "kiali-tls"`
- Components Used: Nginx Ingress Controller 1.9.5 Helm version 4.9.0 Kiali Operator v1.77.0
`/nginx-ingress-controller
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.9.5
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6
Here is the issue which use command line kubectl describe kiali -n istio and ingress of Kiali wasn't created.
Status:
Conditions:
Message:
Reason:
Status: False
Type: Successful
Message: Running reconciliation
Reason: Running
Status: False
Type: Running
Ansible Result:
Changed: 5
Failures: 1
Ok: 64
Skipped: 51
Message: Failed to patch object: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \\"validate.nginx.ingress.kubernetes.io\\" denied the request: annotation group ConfigurationSnippet contains risky annotation based on ingress configuration","reason":"BadRequest","code":400}\n'
Reason: Failed
Status: True
Type: Failure
Deployment:
Instance Name: kiali
Namespace: istio
Environment:
Is Kubernetes: true
Kubernetes Version: 1.24.17
Operator Version: v1.77.0
Progress:
Duration: 0:00:26
Message: 5. Creating core resources
Spec Version: default
Events: <none>
But when set to Critical or delete annotations-risk-level in configmap, the issue was gone and working fine.
Just got stuck when set to High, Medium or Low.
So please kindly for take a look.
Thank you guys.
I also encountered the same problem, when I enabled allow-snippet-annotations in the ingress-nginx-controller configmap.
apiVersion: v1
data:
allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":null,"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.12.0-beta.0"},"name":"ingress-nginx-controller","namespace":"ingress-nginx"}}
creationTimestamp: "2024-11-16T13:13:31Z"
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.12.0-beta.0
name: ingress-nginx-controller
namespace: ingress-nginx
resourceVersion: "2647510"
uid: 5c241c30-f3a1-432c-8c0b-b03235941dba
Create an ingress as follows:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.provider: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header "X-B3-Sampled" "1";
name: gitlab
namespace: gitlab
spec:
rules:
- host: gitlab.test.ai
http:
paths:
- backend:
service:
name: gitlab-http-svc
port:
number: 80
path: /
pathType: Prefix
tls:
- secretName: test-tls-secret
Error when creating:
Warning: annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
Error from server (BadRequest): error when creating "ingress-deploy.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: annotation group ConfigurationSnippet contains risky annotation based on ingress configuration
For the original issue description
% k run nginx --image nginx:alpine --port 80
pod/nginx created
[~]
% k get po
NAME READY STATUS RESTARTS AGE
nginx 0/1 ContainerCreating 0 3s
[~]
% k expose pod nginx --name 300
The Service "300" is invalid: metadata.name: Invalid value: "300": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')
[~]
@shaxiaozz there is no such service name in your message
/close
@longwuyuan: Closing this issue.
In response to this:
For the original issue description
% k run nginx --image nginx:alpine --port 80 pod/nginx created [~] % k get po NAME READY STATUS RESTARTS AGE nginx 0/1 ContainerCreating 0 3s [~] % k expose pod nginx --name 300 The Service "300" is invalid: metadata.name: Invalid value: "300": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?') [~]@shaxiaozz there is no such service name in your message
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@longwuyuan Hi, what does "there is no such service name in your message" mean?
About admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: annotation group ConfigurationSnippet contains risky annotation based on ingress configurationreported an error, I think it should be the configmap configuration, sorry ~. If you want to configure Configurationsnippet in Annotations, annotations-risk-level should be Critical
For details, please refer to: https://kubernetes.github.io/ingress-nginx/usel-nginx- configuration/annotations-risk/
So my configmap is:
apiVersion: v1
data:
allow-snippet-annotations: "true"
annotations-risk-level: Critical
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.12.0-beta.0
name: ingress-nginx-controller
namespace: ingress-nginx
-
@shaxiaozz The first post here is the issue description.
-
The issue description of this issue shows this line
nginx.ingress.kubernetes.io/proxy-next-upstream: "300"
-
The error shown in the issue description is caused by using a name like
"300"for a service -
I have replicated the same error in my test and posted it above and copy/pasted below also
% k run nginx --image nginx:alpine --port 80
pod/nginx created
[~]
% k get po
NAME READY STATUS RESTARTS AGE
nginx 0/1 ContainerCreating 0 3s
[~]
% k expose pod nginx --name 300
The Service "300" is invalid: metadata.name: Invalid value: "300": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')
[~]
-
None of you post shows a service named
"300"or similar all numeric name of a service. So the original issue and your message are NOT about the same problem. Your problem is unrelated to this issue. -
So please do not keep posting here because there is lack of resources. Instead discuss your issue at https://kubernetes.slack.com in the ingress-nginx-users channel. Register at https://slack.k8s.io, if needed.
@srikiz / @longwuyuan / @shaxiaozz
Any way to add this breaking change to https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.12.0 ?
That annotations-risk-level: Critical is now annotations-risk-level: High. The change made most of our ingresses not be served anymore and we took some time to figure that it is because of a configuration-snippet which has a risk level of Critical.
Best regards
Just an FYI - I originally created a bad annotation nginx.ingress.kubernetes.io/proxy-next-upstream: "300" just to reproduce the problem where a bad annotation resulted in a bad ingress configuration when enableAnnotationValidation flag was set to true. But the same validation would work with the validating webhook rejecting the request without specifying the enableAnnotationValidation flag.
However, now it looks like with the new https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.11.5 release, the validation of the generated NGINX configuration is now disabled to fix CVE-2025-1974 and it would mean a user specifying a bad annotation like above (though it doesn't make sense to have a service named 300, but can be created by a user by mistake) would result in a bad ingress configuration, and we would need to watch out for the line of dashes followed by "Error:" if case such an error happens.
I haven't tried enabling the annotation validation flag yet in the config map but plan to test that in some time.