cilium-cli icon indicating copy to clipboard operation
cilium-cli copied to clipboard

workflows: add missing Relay port-forward to external workloads testing

Open nbusseneau opened this issue 2 years ago â€ĸ 2 comments

We somehow missed that one, resulting in flows not being validated in connectivity test via the external workloads workflow.

nbusseneau avatar Jul 20 '21 19:07 nbusseneau

It looks like flow validation completely fails on external workload connectivity test: https://github.com/cilium/cilium-cli/actions/runs/1050143318

2021-07-20T20:07:41.307935214Z 📋 Test Report
2021-07-20T20:07:41.307984955Z ❌ 11/11 tests failed (11/11 actions), 0 tests skipped, 0 scenarios skipped:
2021-07-20T20:07:41.308019492Z Test [no-policies]:
2021-07-20T20:07:41.308045282Z   ❌ no-policies/pod-to-remote-nodeport/curl-0: cilium-test/client-7b7bf54b85-gvhrd (10.104.0.157) -> cilium-test/echo-other-node (echo-other-node:8080)
2021-07-20T20:07:41.308499106Z Test [dns-only]:
2021-07-20T20:07:41.308519083Z   ❌ dns-only/pod-to-pod/curl-0: cilium-test/client-7b7bf54b85-gvhrd (10.104.0.157) -> cilium-test/echo-other-node-697d5d69b7-dqcl2 (10.104.1.245:8080)
2021-07-20T20:07:41.308550811Z Test [client-egress]:
2021-07-20T20:07:41.308559053Z   ❌ client-egress/pod-to-pod/curl-0: cilium-test/client2-666976c95b-rfjbr (10.104.1.13) -> cilium-test/echo-other-node-697d5d69b7-dqcl2 (10.104.1.245:8080)
2021-07-20T20:07:41.308566409Z Test [to-entities-world]:
2021-07-20T20:07:41.308573303Z   ❌ to-entities-world/pod-to-world/https-to-cilium-io-0: cilium-test/client-7b7bf54b85-gvhrd (10.104.0.157) -> cilium-io-https (cilium.io:443)
2021-07-20T20:07:41.308579848Z Test [echo-ingress-l7]:
2021-07-20T20:07:41.308586550Z   ❌ echo-ingress-l7/pod-to-pod/curl-0: cilium-test/client-7b7bf54b85-gvhrd (10.104.0.157) -> cilium-test/echo-other-node-697d5d69b7-dqcl2 (10.104.1.245:8080)
2021-07-20T20:07:41.308594751Z Test [client-egress-l7]:
2021-07-20T20:07:41.308628289Z   ❌ client-egress-l7/pod-to-world/https-to-cilium-io-0: cilium-test/client-7b7bf54b85-gvhrd (10.104.0.157) -> cilium-io-https (cilium.io:443)
2021-07-20T20:07:41.308635989Z Test [allow-all]:
2021-07-20T20:07:41.308642578Z   ❌ allow-all/pod-to-remote-nodeport/curl-0: cilium-test/client2-666976c95b-rfjbr (10.104.1.13) -> cilium-test/echo-other-node (echo-other-node:8080)
2021-07-20T20:07:41.308664932Z Test [client-ingress]:
2021-07-20T20:07:41.308672726Z   ❌ client-ingress/client-to-client/ping-0: cilium-test/client-7b7bf54b85-gvhrd (10.104.0.157) -> cilium-test/client2-666976c95b-rfjbr (10.104.1.13:0)
2021-07-20T20:07:41.308678565Z Test [echo-ingress]:
2021-07-20T20:07:41.308723785Z Connectivity test failed: 11 tests failed
2021-07-20T20:07:41.308684773Z   ❌ echo-ingress/pod-to-pod/curl-0: cilium-test/client-7b7bf54b85-gvhrd (10.104.0.157) -> cilium-test/echo-same-node-7967996674-v9j2f (10.104.0.246:8080)
2021-07-20T20:07:41.308748855Z Test [to-fqdns]:
2021-07-20T20:07:41.308755205Z   ❌ to-fqdns/pod-to-world/https-to-cilium-io-0: cilium-test/client-7b7bf54b85-gvhrd (10.104.0.157) -> cilium-io-https (cilium.io:443)
2021-07-20T20:07:41.308762489Z Test [to-cidr-1111]:
2021-07-20T20:07:41.308781245Z   ❌ to-cidr-1111/pod-to-cidr/cloudflare-1001-0: cilium-test/client-7b7bf54b85-gvhrd (10.104.0.157) -> cloudflare-1001 (1.0.0.1:80)

However I see absolutely no flows being printed despite the --debug --all-flows flags being present. Hubble initially looks good:

2021-07-20T20:05:18.315523652Z 🔭 Enabling Hubble telescope...
2021-07-20T20:05:18.337100405Z ℹī¸  Hubble is OK, flows: 8190/8190

But then fails:

2021-07-20T20:05:30.008818031Z   đŸ”Ĩ Timeout waiting for flow listener to become ready
2021-07-20T20:05:30.008935933Z   đŸ”Ĩ Receiving flows from Hubble Relay: hubble server status failure: context canceled

Are we actually expecting flows to work with external workloads, or is it actually normal that Hubble was never enabled all the way through for external workloads testing? If the latter, I will update this PR to simply remove the Relay enable bits as they are unneeded.

nbusseneau avatar Jul 20 '21 20:07 nbusseneau

I'm putting this back in draft until we have more information. Paging @tklauser @jrajahalme in case they have intel :)

nbusseneau avatar Jul 26 '21 14:07 nbusseneau