external-dns icon indicating copy to clipboard operation
external-dns copied to clipboard

--namespace still trying to read various cluster scope

Open wasabii opened this issue 7 months ago • 1 comments
trafficstars

My install comes off the Helm chart you guys provide, which sets --namespace when namespaced: true.

The Chart then creates a RoleBinding instead of a ClusterRoleBinding. Which makes sense.

However the service itself:

Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User \"system:serviceaccount:demoapp:demoapp-7xhgd-external-dns-private\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError"

Failed to watch *v1beta1.Gateway: failed to list *v1beta1.Gateway: gateways.gateway.networking.k8s.io is forbidden: User \"system:serviceaccount:demoapp:demoapp-7xhgd-external-dns-private\" cannot list resource \"gat ││ eways\" in API group \"gateway.networking.k8s.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError"

Seems it is trying to watch resources at the cluster scope which it would not have accecss to.

wasabii avatar Apr 15 '25 23:04 wasabii

Share all arguments, non helm

ivankatliarchuk avatar May 16 '25 13:05 ivankatliarchuk

@ivankatliarchuk

I can confirm that I've reproduced the problem in my environment as well.

Problem Confirmation

When namespaced: true is set with Gateway API sources (e.g., gateway-httproute), the Helm chart correctly creates a Role instead of ClusterRole. However, the gatewayRouteSource code attempts to list all namespaces via nsInformer.Lister().List(), which fails with "namespaces is forbidden" error due to lack of cluster-wide namespace permissions.

Proposed Solutions

I see two potential approaches to fix this:

Approach 1: Split RBAC (Implemented & Tested ✅)

  • Create an additional ClusterRole specifically for namespace access when namespaced: true AND Gateway sources are used
  • Modify Helm templates to bind this ClusterRole alongside the namespace-scoped Role
  • Status: I have implemented and tested this approach successfully. (I haven't opened the PR yet)

Approach 2: Eliminate Namespace List Requirement (Preferred)

  • Modify the gateway source implementation to only access the specific namespaces defined in config (Namespace and GatewayNamespace)
  • Remove the dependency on cluster-wide namespace listing entirely
  • Status: Not yet implemented, but I believe this is the better approach

My Recommendation

I personally favor Approach 2 for the following reasons:

  1. Security: When namespaced: true is specified, the deployment should not require cluster-wide access to other namespaces
  2. Principle of least privilege: More restrictive permissions align better with the intent of namespaced deployments
  3. Cleaner architecture: Eliminates unnecessary namespace informer complexity

Next Steps

I'm prepared to create a PR once we align on the preferred approach. Both solutions are viable, but I'd appreciate your thoughts on which direction would be most suitable for the project.

What are your thoughts on these approaches?

u-kai avatar Jun 23 '25 12:06 u-kai

Option 2 sounds the way to go

I think the problem is sligthly more complex, but hard to say.

https://github.com/kubernetes-sigs/external-dns/blob/e324da8a43d862f9c7b798ee0a0d236a26b72ebe/source/gateway.go#L149, replace with

kubeInformerFactory := kubeinformers.NewSharedInformerFactoryWithOptions(kubeClient, 0, kubeinformers.WithNamespace(namespace))

The line

namespaces, err := src.nsInformer.Lister().List(labels.Everything())

Does not support namespace, as it's a namespace informer. What we could do, not sure if needed actually is

if namespace != ""
  ... ....List(labels.Everything())
else:
  src.nsInformer.Lister().Get(namespace)

^ but this could trigger/reveal some other issues, so not too sure

ivankatliarchuk avatar Jun 24 '25 06:06 ivankatliarchuk

@ivankatliarchuk Thanks for the reply! Understood that option 2 is preferred. You're right — it may be a bit more complex, so I'll investigate it further. If I can get it working, I'll open a PR. Thanks again!

u-kai avatar Jun 24 '25 12:06 u-kai

@ivankatliarchuk

After investigating the implementation in detail, I've concluded that Option 1 (Split RBAC) is the most feasible approach.
The key finding is that NamespacesFromSelector in Gateway API requires access to namespace information for proper route validation. This makes eliminating the namespace informer (Option 2) technically impossible while maintaining full Gateway API compliance.

Additionally, I discovered that both List and Get operations on namespace resources require ClusterRole permissions when accessing namespaces outside the current scope. Since Gateway API allows cross-namespace references between Gateways and Routes, we inevitably need cluster-wide namespace access regardless of the specific API method used.

I apologize for initially suggesting Option 2 as viable.

I'm now preparing to submit a PR with the enhanced ClusterRole and Role additions to fully resolve this issue.

u-kai avatar Jun 25 '25 13:06 u-kai

Am I correct that this RBAC permission is not ACTUALLY needed unless a user actually does specify a gateway selector with a namespace?

If so, why would external-DNS fail if the user does no such thing?

wasabii avatar Oct 25 '25 19:10 wasabii