Enable Cilium for AKS (Advanced Container Networking)

[emirgens]

Enable in dev first

Add in Terraform AKS enable add-on

• Cilium
• Advanced Container Networking

TODO:

• Ask Microsoft if Cilium is supported on ARM or not, ref. https://learn.microsoft.com/en-us/azure/aks/azure-cni-overlay?tabs=kubectl#upgrade-an-existing-cluster-to-cni-overlay Not an issue

DoD Ready to be enabled in Prod clusters

[Richard87]

https://learn.microsoft.com/en-us/azure/aks/azure-cni-powered-by-cilium

Started cililum-26 with these network options:

    AKS_NETWORK_OPTIONS=(
        --network-plugin "azure"
        --network-plugin-mode overlay
        --network-dataplane cilium
    )

Setup Advanced Networking with managed Cilium, but bring your own Grafana/Prometheus

https://learn.microsoft.com/en-us/azure/aks/advanced-network-observability-bring-your-own-cli?tabs=non-cilium

az aks update --resource-group clusters-dev --name cilium-26 --enable-advanced-network-observability
...
k get pods -n kube-system -l k8s-app=hubble-relay
# NAME                            READY   STATUS    RESTARTS   AGE
# hubble-relay-55b65f695c-6bnwk   1/1     Running   0          4m9s
...
k port-forward -n kube-system svc/hubble-relay --address 127.0.0.1 4245:443

Level 7 / DNS & HTTP visiblity:

https://docs.cilium.io/en/latest/observability/visibility/#layer-7-protocol-visibility

Note: We should enable --hubble-redact-enabled to redact sensitive http data like query/headers/auth cookies etc

[Richard87]

To upgrade existing Calico cluster to Cilium:

• Remove Calico and network policies
• Upgrade network mode to overlay
• Install cilium