Can't install two instances of teleport cluster onto same k8s

[gecube]

Good day !

I am researching the way how to provide teleport service to my multiple tenants (developer teams). For the sake of strong isolation I want to install several instances of teleport cluster with different external endpoints. But I am facing the issue that teleport proxy is authenticating against teleport-auth so auth service needs ClusterRole to work with TokenReview API of k8s. But ClusterRole name is hardcoded in Helm Chart. I want to mention that it is completely correct to have a several namespaces like teleport-cluster-1, teleport-cluster-2 ... and install the same RELEASE name into different namespaces. But the issue with the cluster global objects arises. Like:

Helm install failed: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "teleport" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "teleport-cluster-2": current value is "teleport-cluster"

It happens because of https://github.com/gravitational/teleport/blob/1e1ecef9e894e070554c81cf116911470f953e2a/examples/chart/teleport-cluster/templates/auth/clusterrolebinding.yaml#L5 and https://github.com/gravitational/teleport/blob/1e1ecef9e894e070554c81cf116911470f953e2a/examples/chart/teleport-cluster/templates/auth/clusterrole.yaml#L5

I am proposing to change the ClusterRole and ClusterRoleBinding from {{ .Release.Name }} to something more sane like {{ .Release.Namespace }}-{{ .Release.Name }}

Please share your ideas on this topic.

[hugoShaka]

THis is a common issue and a sane suggestion. However, this would be a breaking change for everyone who disabled rbac creation and manually created the bindings (basically every large corp with a security team separately managing RBAC). I don't think we want to break existing setups.

cc @webvictim WDYT?

[gaalw]

@hugoShaka

However, this would be a breaking change for everyone who disabled rbac creation and manually created the bindings (basically every large corp with a security team separately managing RBAC). I don't think we want to break existing setups.

Why ? I can argue that the name of service account is the same, so effectively meaning that who is managing the RBAC externally won't suffer from the change.

[hugoShaka]

I think you're right. I was considering the case where the ClusterRole name would change and it would break theClusterRoleBinding, but both are gated by the rbac.enabled value of the chart, so this is not possible for a user to use our role without the rolebinding (or if they do, they are doing weird things we don't want to support).

Thank you for the heads up.

[hugoShaka]

Starting with v15, teleport-cluster will support multiple releases with the same name in different namespaces as long as the operator CRD installation is disabled (or a fortiori the operator is disabled).

[programmerq]

@hugoShaka I see that #35862 addresses the CRB. To really make this work, the ClusterRole would also need the same treatment.

https://github.com/gravitational/teleport/blob/v15.3.1/examples/chart/teleport-cluster/templates/auth/clusterrole.yaml#L6

I ran into someone that got bit by this discrepancy. They had rbac.create:false in one cluster, and enabled in another. The one with it disabled didn't get a CRB specific to it, so it failed.

[hugoShaka]

I think that changing the default ClusterRole name to contain the release namespace would be a breaking change for users with rbac.create: false. If it's not I can rename the cluster role as well, but I fear this will cause more issues.

[webvictim]

Yes, if the chart isn't managing the creation of the role then this change would potentially break things for any user who created their ClusterRole based on the old naming scheme.

The only thing we could do would be to force the old naming scheme (i.e. release name only) for anyone with rbac.create set to false, which would prevent them from having multiple chart installations in the same cluster without manual workarounds (e.g. kustomize) but it would also enable it for everyone else who allows the chart to manage their RBAC. This might be an acceptable compromise?

[gecube]

@webvictim Hi! Thanks for finding. I did not carefully think about this scenario. @programmerq Hi! Did somebody complain on this change? I see that we already introduced it, so it is always difficult to turn the minced meat back because somebody can rely on the new behaviour. @webvictim hm... maybe makes sense.

[programmerq]

If the chart adopts a similar paradigm for auth/proxy specific resources as was done in the v12 chart refactor, there might be a good path forward.

If proxy.rbac.create and auth.rbac.create are populated by rbac.create only if unset, I don't think it'd break anything.

On Thu, May 9, 2024, 9:51 PM Gus Luxton @.***> wrote:

Yes, if the chart isn't managing the creation of the role then this change would potentially break things for any user who created their ClusterRole based on the old naming scheme.

The only thing we could do would be to force the old naming scheme (i.e. release name only) for anyone with rbac.create set to false, which would prevent them from having multiple chart installations in the same cluster without manual workarounds - but it would also enable it for everyone else who allows the chart to manage their RBAC. This might be an acceptable compromise?

— Reply to this email directly, view it on GitHub https://github.com/gravitational/teleport/issues/35486#issuecomment-2103706258, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAJDHUKADKS2ZEA6JBMF5DZBQRZBAVCNFSM6AAAAABALGHI2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBTG4YDMMRVHA . You are receiving this because you modified the open/close state.Message ID: @.***>