Tests matching `kube-dns` in their policy fail with coredns `hostNetwork:true`

[doniacld]

Bug report

General Information

• Cilium CLI version (run cilium version)

cilium version                                                                                 
cilium-cli: enterprise/cilium-cli/v0.15.6-cee.1-177-gd19d9d9ccc compiled with go1.20.2 on darwin/arm64
cilium image (default): v1.14.0
cilium image (stable): v1.14.2
cilium image (running): unknown. Unable to obtain cilium version, no cilium pods found in namespace "kube-system"

level=info msg="Cilium 1.13.7-cee.1 36083a04 2023-09-14T08:21:34+00:00 go version go1.20.8 linux/arm64" subsys=daemon

• Orchestration system version in use (e.g. kubectl version, ...)

 kubectl version --short                                                                      
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

• Platform / infrastructure information (e.g. AWS / Azure / GCP, image / kernel versions) Running on kind

• Link to relevant artifacts (policies, deployments scripts, ...) Using make kind in cilium/cilium

• Generate and upload a system zip: cilium sysdump cilium-sysdump-20230929-161915.zip

• Output of the test

📋 Test Report
❌ 1/1 tests failed (8/8 actions), 54 tests skipped, 0 scenarios skipped:
Test [to-fqdns]:
  ❌ to-fqdns/pod-to-world-2/https-cilium-io-0: cilium-test/client-6b4b857d98-ktcw9 (10.0.0.240) -> cilium-io-https (cilium.io:443)
  ❌ to-fqdns/pod-to-world-2/https-cilium-io-1: cilium-test/client2-646b88fb9b-2cs5w (10.0.0.20) -> cilium-io-https (cilium.io:443)
  ❌ to-fqdns/pod-to-world/http-to-one.one.one.one-0: cilium-test/client-6b4b857d98-ktcw9 (10.0.0.240) -> one.one.one.one-http (one.one.one.one:80)
  ❌ to-fqdns/pod-to-world/https-to-one.one.one.one-0: cilium-test/client-6b4b857d98-ktcw9 (10.0.0.240) -> one.one.one.one-https (one.one.one.one:443)
  ❌ to-fqdns/pod-to-world/https-to-one.one.one.one-index-0: cilium-test/client-6b4b857d98-ktcw9 (10.0.0.240) -> one.one.one.one-https-index (one.one.one.one:443)
  ❌ to-fqdns/pod-to-world/http-to-one.one.one.one-1: cilium-test/client2-646b88fb9b-2cs5w (10.0.0.20) -> one.one.one.one-http (one.one.one.one:80)
  ❌ to-fqdns/pod-to-world/https-to-one.one.one.one-1: cilium-test/client2-646b88fb9b-2cs5w (10.0.0.20) -> one.one.one.one-https (one.one.one.one:443)
  ❌ to-fqdns/pod-to-world/https-to-one.one.one.one-index-1: cilium-test/client2-646b88fb9b-2cs5w (10.0.0.20) -> one.one.one.one-https-index (one.one.one.one:443)
connectivity test failed: 1 tests failed

Hubble flows

Sep 28 13:30:15.646: cilium-test/client-6b4b857d98-ktcw9:58967 (ID:13038) <> 172.19.0.2:53 (kube-apiserver) policy-verdict:none EGRESS DENIED (UDP)
Sep 28 13:30:15.646: cilium-test/client-6b4b857d98-ktcw9:58967 (ID:13038) <> 172.19.0.2:53 (kube-apiserver) Policy denied DROPPED (UDP)
Sep 28 13:30:15.646: cilium-test/client-6b4b857d98-ktcw9:58967 (ID:13038) <> 172.19.0.2:53 (kube-apiserver) policy-verdict:none EGRESS DENIED (UDP)
Sep 28 13:30:15.646: cilium-test/client-6b4b857d98-ktcw9:58967 (ID:13038) <> 172.19.0.2:53 (kube-apiserver) Policy denied DROPPED (UDP)

How to reproduce the issue

1. Deploy a kind cluster
2. Update coredns to use host network interface.

 kubectl -n kube-system patch deploy/coredns -p '{"spec":{"template":{"spec":{"hostNetwork":true}}}}'

2. Launch cilium-cli test to-fqdns

./cilium connectivity test --namespace kube-system --test to-fqdns --debug

Explanation of the failure

“to-fqdn” tests running nslookup command would fail with core-dns setup as hostNetwork:true because core-dns will not be seen as a Cilium-managed pod but as a remote-node.

If the core-dns is set up hostNetwork:true, we could update the policy using toEntities selector to allow egress traffic on remote-node

• Additional information: we cannot use a different selector than toEntities:
  • toServices if used without selectors, it does not make sense since core-dns is not a service. In the case it would sepcify a selector, it is using toCIDR under the hood which supports only pod selection. (NB: there is a WIP to support node selection but not sure if it would work anyway)
  • toEndpoints it’s not an endpoint since it’s not seen as cilium-managed endpoint.