kubernetes-ingress-controller
kubernetes-ingress-controller copied to clipboard
Kong
testing: run Ingress conformance test
Ingress conformance is currently being worked on: https://github.com/kubernetes-sigs/ingress-controller-conformance
The scope here is to add conformance tests in the CI process to highlight and flag issues during development and ensure conformance to k8s specs.
Acceptance criteria
- [ ] A manual conformance test run against a recent KIC version. If any test failures occur, having these documented as GitHub issues.
- [ ] Each KIC PR use the tool ingress-controller-conformance run the conformance tests. If too much load, run per release.
For now I've created a PR to make the project build again: https://github.com/kubernetes-sigs/ingress-controller-conformance/pull/94
So after looking at the results (that I can post later from a full test run) it seems that KIC doesn't fully pass those.
Given that the upstream project is not actively maintained (last commits in September 2021) In that case I believe we can either:
- retrofit some upstream tests that our ingress controller passes into our tests suite in this repo which will require some effort to add the boilerplate around it, and later if https://github.com/kubernetes-sigs/ingress-controller-conformance picks up steam work on making all upstream tests pass with KIC
- fork the upstream project and work on test cases but then we're diverging from upstream and whole point is to have 1 set of "conformance" tests for ingress controllers
Can you enumerate which tests are not passing, and what changes would be needed to become conformant?
This is the test output that I ran against GKE 1.23 ( I ran this on ARM64 box with this patch so that the project builds and runs without errors: https://github.com/kubernetes-sigs/ingress-controller-conformance/pull/94)
...and what changes would be needed to become conformant?
As for that unfortunately I don't know at this point.
$ ./ingress-controller-conformance -ingress-class kong -wait-time-for-ingress-status 120s
Feature: Ingress class
Ingresses can be implemented by different controllers, often with different configuration.
Each Ingress definition could specify a class, a reference to an IngressClass resource that contains
additional configuration including the name of the controller that should implement the class.
https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class
Scenario: An Ingress with an invalid ingress class should not send traffic to the matching backend service # features/ingress_class.feature:9
Given an Ingress resource in a new random namespace # feature.go:51 -> anIngressResourceInANewRandomNamespace
"""
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-ingress-class
spec:
ingressClassName: some-invalid-class-name
rules:
- host: "ingress-class"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ingress-class-prefix
port:
number: 8080
"""
Then The Ingress status should not contain the IP address or FQDN # feature.go:79 -> theIngressStatusShouldNotContainTheIPAddressOrFQDN
1 scenarios (1 passed)
2 steps (2 passed)
2m5.72046475s
Feature: Load Balancing
An Ingress exposing a backend service with multiple replicas should use all the pods available
The feature sessionAffinity is not configured in the backend service https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#service-v1-core
Background:
Given a new random namespace # feature.go:61 -> aNewRandomNamespace
Given an Ingress resource named "load-balancing" with this spec: # feature.go:71 -> anIngressResourceNamedWithThisSpec
"""
defaultBackend:
service:
name: echo-service
port:
number: 8080
"""
Then The Ingress status shows the IP address or FQDN where it is exposed # feature.go:92 -> theIngressStatusShowsTheIPAddressOrFQDNWhereItIsExposed
waiting for ingress status update: timed out waiting for the condition
Then The backend deployment "echo-service" for the ingress resource is scaled to 10 # feature.go:137 -> theBackendDeploymentForTheIngressResourceIsScaledTo
Scenario Outline: An Ingress with no rules should send all requests to the default backend and # features/load_balancing.feature:19
When I send 100 requests to "http://load-balancing" # feature.go:102 -> iSendRequestsTo
Then all the responses status-code must be 200 and the response body should contain the IP address of 10 different Kubernetes pods # feature.go:124 -> allTheResponsesStatuscodeMustBeAndTheResponseBodyShouldContainTheIPAddressOfDifferentKubernetesPods
--- Failed steps:
Scenario Outline: An Ingress with no rules should send all requests to the default backend and # features/load_balancing.feature:19
Then The Ingress status shows the IP address or FQDN where it is exposed # features/load_balancing.feature:16
Error: waiting for ingress status update: timed out waiting for the condition
1 scenarios (1 failed)
6 steps (2 passed, 1 failed, 3 skipped)
2m5.656539959s
Feature: Default backend
An Ingress with no rules sends all traffic to the single default backend.
The default backend is part of the Ingress resource spec field `defaultBackend`.
If none of the hosts or paths match the HTTP request in the
Ingress objects, the traffic is routed to your default backend.
Background:
Given a new random namespace # feature.go:61 -> aNewRandomNamespace
Given an Ingress resource named "default-backend" with this spec: # feature.go:71 -> anIngressResourceNamedWithThisSpec
"""
defaultBackend:
service:
name: echo-service
port:
number: 8080
"""
Then The Ingress status shows the IP address or FQDN where it is exposed # feature.go:92 -> theIngressStatusShowsTheIPAddressOrFQDNWhereItIsExposed
waiting for ingress status update: timed out waiting for the condition
Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
Given a new random namespace # feature.go:61 -> aNewRandomNamespace
Given an Ingress resource named "default-backend" with this spec: # feature.go:71 -> anIngressResourceNamedWithThisSpec
"""
defaultBackend:
service:
name: echo-service
port:
number: 8080
"""
Then The Ingress status shows the IP address or FQDN where it is exposed # feature.go:92 -> theIngressStatusShowsTheIPAddressOrFQDNWhereItIsExposed
When I send a "<method>" request to http://"<host>"/"<path>" # feature.go:102 -> iSendARequestToHttp
Then the response status-code must be 200 # feature.go:106 -> theResponseStatuscodeMustBe
And the response must be served by the "echo-service" service # feature.go:110 -> theResponseMustBeServedByTheService
And the response proto must be "HTTP/1.1" # feature.go:114 -> theResponseProtoMustBe
And the response headers must contain <key> with matching <value> # feature.go:118 -> theResponseHeadersMustContainKeyWithMatchingValue
| key | value |
| Content-Length | * |
| Content-Type | * |
| Date | * |
| Server | * |
And the request method must be "<method>" # feature.go:122 -> theRequestMethodMustBe
And the request path must be "<path>" # feature.go:126 -> theRequestPathMustBe
And the request proto must be "HTTP/1.1" # feature.go:130 -> theRequestProtoMustBe
And the request headers must contain <key> with matching <value> # feature.go:134 -> theRequestHeadersMustContainKeyWithMatchingValue
| key | value |
| User-Agent | Go-http-client/1.1 |
Examples:
| method | host | path |
| GET | my-host | |
waiting for ingress status update: timed out waiting for the condition
| GET | my-host | sub-path |
waiting for ingress status update: timed out waiting for the condition
| POST | some-host | |
waiting for ingress status update: timed out waiting for the condition
| PUT | | resource |
waiting for ingress status update: timed out waiting for the condition
| DELETE | some-host | resource |
waiting for ingress status update: timed out waiting for the condition
| PATCH | my-host | resource |
waiting for ingress status update: timed out waiting for the condition
--- Failed steps:
Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
Then The Ingress status shows the IP address or FQDN where it is exposed # features/default_backend.feature:19
Error: waiting for ingress status update: timed out waiting for the condition
Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
Then The Ingress status shows the IP address or FQDN where it is exposed # features/default_backend.feature:19
Error: waiting for ingress status update: timed out waiting for the condition
Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
Then The Ingress status shows the IP address or FQDN where it is exposed # features/default_backend.feature:19
Error: waiting for ingress status update: timed out waiting for the condition
Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
Then The Ingress status shows the IP address or FQDN where it is exposed # features/default_backend.feature:19
Error: waiting for ingress status update: timed out waiting for the condition
Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
Then The Ingress status shows the IP address or FQDN where it is exposed # features/default_backend.feature:19
Error: waiting for ingress status update: timed out waiting for the condition
Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
Then The Ingress status shows the IP address or FQDN where it is exposed # features/default_backend.feature:19
Error: waiting for ingress status update: timed out waiting for the condition
6 scenarios (6 failed)
72 steps (12 passed, 6 failed, 54 skipped)
12m35.103788667s
Feature: Host rules
An Ingress may define routing rules based on the request host.
If the HTTP request host matches one of the hosts in the
Ingress objects, the traffic is routed to its backend service.
Background:
Given a new random namespace # feature.go:58 -> aNewRandomNamespace
Given a self-signed TLS secret named "conformance-tls" for the "foo.bar.com" hostname # feature.go:89 -> aSelfsignedTLSSecretNamedForTheHostname
Given an Ingress resource # feature.go:68 -> anIngressResource
"""
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: host-rules
spec:
tls:
- hosts:
- foo.bar.com
secretName: conformance-tls
rules:
- host: "*.foo.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: wildcard-foo-com
port:
number: 8080
- host: foo.bar.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: foo-bar-com
port:
name: http
"""
Then The Ingress status shows the IP address or FQDN where it is exposed # feature.go:101 -> theIngressStatusShowsTheIPAddressOrFQDNWhereItIsExposed
Scenario: An Ingress with a host rule should send TLS traffic to the matching backend service # features/host_rules.feature:47
When I send a "GET" request to "https://foo.bar.com" # feature.go:112 -> iSendARequestTo
Then the secure connection must verify the "foo.bar.com" hostname # feature.go:120 -> theSecureConnectionMustVerifyTheHostname
And the response status-code must be 200 # feature.go:134 -> theResponseStatuscodeMustBe
And the response must be served by the "foo-bar-com" service # feature.go:138 -> theResponseMustBeServedByTheService
And the request host must be "foo.bar.com" # feature.go:142 -> theRequestHostMustBe
Scenario: An Ingress with a host rule should send traffic to the matching backend service # features/host_rules.feature:56
When I send a "GET" request to "http://foo.bar.com" # feature.go:112 -> iSendARequestTo
And the response status-code must be 200 # feature.go:134 -> theResponseStatuscodeMustBe
And the response must be served by the "foo-bar-com" service # feature.go:138 -> theResponseMustBeServedByTheService
And the request host must be "foo.bar.com" # feature.go:142 -> theRequestHostMustBe
Scenario: An Ingress with a host rule should not route traffic when hostname does not match # features/host_rules.feature:64
When I send a "GET" request to "http://subdomain.bar.com" # feature.go:112 -> iSendARequestTo
Then the response status-code must be 404 # feature.go:134 -> theResponseStatuscodeMustBe
Scenario: An Ingress with a wildcard host rule should send traffic to the matching backend service # features/host_rules.feature:70
When I send a "GET" request to "http://bar.foo.com" # feature.go:112 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:134 -> theResponseStatuscodeMustBe
And the response must be served by the "wildcard-foo-com" service # feature.go:138 -> theResponseMustBeServedByTheService
And the request host must be "bar.foo.com" # feature.go:142 -> theRequestHostMustBe
Scenario: An Ingress with a wildcard host rule should not route traffic matching on more than a single dns label # features/host_rules.feature:78
When I send a "GET" request to "http://baz.bar.foo.com" # feature.go:112 -> iSendARequestTo
Then the response status-code must be 404 # feature.go:134 -> theResponseStatuscodeMustBe
expected status code 404 but 200 was returned
Scenario: An Ingress with a wildcard host rule should not route traffic matching no dns label # features/host_rules.feature:84
When I send a "GET" request to "http://foo.com" # feature.go:112 -> iSendARequestTo
Then the response status-code must be 404 # feature.go:134 -> theResponseStatuscodeMustBe
--- Failed steps:
Scenario: An Ingress with a wildcard host rule should not route traffic matching on more than a single dns label # features/host_rules.feature:78
Then the response status-code must be 404 # features/host_rules.feature:82
Error: expected status code 404 but 200 was returned
6 scenarios (5 passed, 1 failed)
43 steps (42 passed, 1 failed)
1m38.584221125s
Feature: Path rules
An Ingress may define routing rules based on the request path.
If the HTTP request path matches one of the paths in the
Ingress objects, the traffic is routed to its backend service.
Background:
Given an Ingress resource in a new random namespace # feature.go:55 -> anIngressResourceInANewRandomNamespace
"""
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: path-rules
spec:
rules:
- host: "exact-path-rules"
http:
paths:
- path: /foo
pathType: Exact
backend:
service:
name: foo-exact
port:
number: 8080
- host: "prefix-path-rules"
http:
paths:
- path: /foo
pathType: Prefix
backend:
service:
name: foo-prefix
port:
number: 8080
- path: /aaa/bbb
pathType: Prefix
backend:
service:
name: aaa-slash-bbb-prefix
port:
number: 8080
- path: /aaa
pathType: Prefix
backend:
service:
name: aaa-prefix
port:
number: 8080
- host: "mixed-path-rules"
http:
paths:
- path: /foo
pathType: Prefix
backend:
service:
name: foo-prefix
port:
number: 8080
- path: /foo
pathType: Exact
backend:
service:
name: foo-exact
port:
number: 8080
- host: "trailing-slash-path-rules"
http:
paths:
- path: /aaa/bbb/
pathType: Prefix
backend:
service:
name: aaa-slash-bbb-slash-prefix
port:
number: 8080
- path: /foo/
pathType: Exact
backend:
service:
name: foo-slash-exact
port:
number: 8080
"""
Then The Ingress status shows the IP address or FQDN where it is exposed # feature.go:83 -> theIngressStatusShowsTheIPAddressOrFQDNWhereItIsExposed
Scenario: An Ingress with exact path rules should send traffic to the matching backend service # features/path_rules.feature:95
When I send a "GET" request to "http://exact-path-rules/foo" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "foo-exact" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with exact path rules should not match requests with trailing slash # features/path_rules.feature:102
When I send a "GET" request to "http://exact-path-rules/foo/" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 404 # feature.go:102 -> theResponseStatuscodeMustBe
Scenario: An Ingress with exact path rules should be case sensitive # features/path_rules.feature:108
When I send a "GET" request to "http://exact-path-rules/FOO" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 404 # feature.go:102 -> theResponseStatuscodeMustBe
Scenario: An Ingress with exact path rules should not match any other label # features/path_rules.feature:114
When I send a "GET" request to "http://exact-path-rules/bar" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 404 # feature.go:102 -> theResponseStatuscodeMustBe
Scenario: An Ingress with prefix path rules should send traffic to the matching backend service # features/path_rules.feature:120
When I send a "GET" request to "http://prefix-path-rules/foo" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "foo-prefix" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with prefix path rules should ignore the request trailing slash and send traffic to the matching backend service # features/path_rules.feature:127
When I send a "GET" request to "http://prefix-path-rules/foo/" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "foo-prefix" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with prefix path rules should be case sensitive # features/path_rules.feature:134
When I send a "GET" request to "http://prefix-path-rules/FOO" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 404 # feature.go:102 -> theResponseStatuscodeMustBe
Scenario: An Ingress with prefix path rules should match multiple labels, match the longest path, and send traffic to the matching backend service # features/path_rules.feature:140
When I send a "GET" request to "http://prefix-path-rules/aaa/bbb" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "aaa-slash-bbb-prefix" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with prefix path rules should match multiple labels, match the longest path, and subpaths and send traffic to the matching backend service # features/path_rules.feature:147
When I send a "GET" request to "http://prefix-path-rules/aaa/bbb/ccc" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "aaa-slash-bbb-prefix" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with prefix path rules should and send traffic to the matching backend service # features/path_rules.feature:154
When I send a "GET" request to "http://prefix-path-rules/aaa/ccc" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "aaa-prefix" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with prefix path rules should match each labels string prefix # features/path_rules.feature:161
When I send a "GET" request to "http://prefix-path-rules/aaaccc" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 404 # feature.go:102 -> theResponseStatuscodeMustBe
Scenario: An Ingress with prefix path rules should ignore the request trailing slash and send traffic to the matching backend service # features/path_rules.feature:167
When I send a "GET" request to "http://prefix-path-rules/foo/" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "foo-prefix" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with mixed path rules should send traffic to the matching backend service where Exact is preferred # features/path_rules.feature:174
When I send a "GET" request to "http://mixed-path-rules/foo" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "foo-exact" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with a trailing slashes in a prefix path rule should ignore the trailing slash and send traffic to the matching backend service # features/path_rules.feature:181
When I send a "GET" request to "http://trailing-slash-path-rules/aaa/bbb" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "aaa-slash-bbb-slash-prefix" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with a trailing slashes in a prefix path rule should ignore the trailing slash and send traffic to the matching backend service # features/path_rules.feature:188
When I send a "GET" request to "http://trailing-slash-path-rules/aaa/bbb/" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 200 # feature.go:102 -> theResponseStatuscodeMustBe
And the response must be served by the "aaa-slash-bbb-slash-prefix" service # feature.go:106 -> theResponseMustBeServedByTheService
Scenario: An Ingress with a trailing slashes in an exact path rule should not match requests without a trailing slash # features/path_rules.feature:195
When I send a "GET" request to "http://trailing-slash-path-rules/foo" # feature.go:94 -> iSendARequestTo
Then the response status-code must be 404 # feature.go:102 -> theResponseStatuscodeMustBe
16 scenarios (16 passed)
74 steps (74 passed)
10m4.672818125s
--- FAIL: TestSuite (1709.74s)
conformance_test.go:129: at least one step/scenario failed
FAIL
Given that we seem to be at the edge of a fairly sizeable time investment if we want to see this issue through to completion and the fact that you got no response in SIG Networking channel on Kubernetes slack, I suggest that we add an agenda item to the next SIG Network Sync to inquire about the state of the conformance tests and their maintainer-ship and see if the community has any insights about our predicament. From there we can probably get a better idea of how much time this is going to take overall.
Given that we seem to be at the edge of a fairly sizeable time investment if we want to see this issue through to completion and the fact that you got no response in SIG Networking channel on Kubernetes slack, I suggest that we add an agenda item to the next SIG Network Sync to inquire about the state of the conformance tests and their maintainer-ship and see if the community has any insights about our predicament. From there we can probably get a better idea of how much time this is going to take overall.
Thanks! Added to the agenda
After several discussions with the community the common thread appears to be that ingress conformance came too late in the game and never got the attention it deserved. As such it's not actively maintained and it would simply appear to not be worth the time to try to implement it until we're ready to invest significantly in upstream as this would possibly be required. For now I'm closing this as not having any priority at the moment, and we can re-asses later if need be.