Flaky test failure during v2.4.2 release

[pmalek]

During v2.4.2 release preparations we've encountered a test failure in our integration-tests-enterprise-postgres suite

The following tests failed:

• TestHTTPRouteExample
• TestKongIngressEssentials
• TestIngressEssentials

[randmonkey]

Looked at recent e2e tests, there seems to be one remaining flaky case: TestDeployAllInOnePostgresWithMultipleReplicas, the other cases were not failing in recent tests. - -This case is related to leader election, so more effects would be used to check what was wrong in leader election. The comments are nit so releated to this issue, created new issue for flaky e2e tests: #2670

[pmalek]

TestKongIngressEssentials failed again in https://github.com/Kong/kubernetes-ingress-controller/runs/7320947630?check_suite_focus=true

=== CONT  TestKongIngressEssentials
    kongingress_test.go:116: 
        	Error Trace:	/home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/kongingress_test.go:116
        	Error:      	Condition never satisfied
        	Test:       	TestKongIngressEssentials
    kongingress_test.go:87: ensuring that KongIngress minking is cleaned up
    kongingress_test.go:66: ensuring that Ingress resources are cleaned up
    kongingress_test.go:52: cleaning up the service minking
    kongingress_test.go:39: cleaning up the deployment minking

Additionally, during the same run, a failure in TestKongIngressEssentials has been observed:

=== CONT  TestPluginEssentials
    plugin_test.go:161: 
        	Error Trace:	/home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/integration/plugin_test.go:161
        	Error:      	Condition never satisfied
        	Test:       	TestPluginEssentials
    plugin_test.go:171: updating Ingress to use cluster plugin legal
    plugin_test.go:194: validating that clusterplugin legal was successfully configured
time="2022-07-13T12:56:55Z" level=debug msg="parsing kubernetes objects into data-plane configuration"
time="2022-07-13T12:56:55Z" level=debug msg="fetching endpoints" service_name=httpbin service_namespace=47f0eb7f-89dd-4b6c-9323-25afd285a27d service_port="&ServicePort***Name:,Protocol:TCP,Port:80,TargetPort:***0 80 ***NodePort:30466,AppProtocol:nil,***"
time="2022-07-13T12:56:55Z" level=debug msg="found endpoints: [***10.244.0.56 80***]" service_name=httpbin service_namespace=47f0eb7f-89dd-4b6c-9323-25afd285a27d service_port="&ServicePort***Name:,Protocol:TCP,Port:80,TargetPort:***0 80 ***NodePort:30466,AppProtocol:nil,***"
time="2022-07-13T12:56:55Z" level=debug msg="listener missing status information" gateway=kong listener=tls
time="2022-07-13T12:56:55Z" level=error msg="failed to fetch secret" error="Secret 90d7adeb-4e5d-4c95-8553-f8a10151b383/secret-test not found" gateway=kong listener=tls secret_name=secret-test secret_namespace=90d7adeb-4e5d-4c95-8553-f8a10151b383
time="2022-07-13T12:56:55Z" level=error msg="secret reference not allowed by ReferencePolicy" error="<nil>" gateway=kong gateway_namespace=90d7adeb-4e5d-4c95-8553-f8a10151b383 listener=tlsother secret_name=secret2 secret_namespace=fe62d10c-af54-4e4c-89e1-49d6fe8fda62
time="2022-07-13T12:56:55Z" level=debug msg="successfully built data-plane configuration"
time="2022-07-13T12:56:55Z" level=debug msg="converting configuration to deck config"
time="2022-07-13T12:56:55Z" level=debug msg="sending configuration to Kong Admin API"
updating plugin request-termination for route 47f0eb7f-89dd-4b6c-9323-25afd285a27d.httpbin.00  ***
   "config": ***
     "body": null,
     "content_type": null,
     "echo": false,
     "message": null,
-    "status_code": 418,
+    "status_code": 451,
     "trigger": null
   ***
   "enabled": true,
   "id": "eb838820-49f9-4ab5-817b-5fac08290f34",
   "name": "request-termination",
   "protocols": [
     "http",
     "https"
   ],
   "route": ***
     "id": "887d04cc-9936-4a87-972e-b76641c9f8ea",
     "name": "47f0eb7f-89dd-4b6c-9323-25afd285a27d.httpbin.00"
   ***
   "tags": [
     "managed-by-ingress-controller"
   ]
 ***
    plugin_test.go:205: deleting Ingress and waiting for routes to be torn down
--- FAIL: TestKongIngressEssentials (365.69s)

[jrsmroz]

Do we still need this issue now that we have build pulse enabled? I would close it if possible. It does not have acceptance criteria, it's sole purpose was to track failures during a release. We should use build pulse now and open issue for concrete flakes that we recently saw.

[pmalek]

Sure thing. Let's reopen if that flake reoccurs, however it's going to be detected.