Shouldn't consul connect support "consul.hashicorp.com/service-name"?

[whiskeysierra]

Somewhat related to #237.

Question

Consul connect allows to specify the service name using the consul.hashicorp.com/connect-service annotation on a pod. But it has no support to change the consul service name associated with the pod's service account. They have to have the same name, as stated:

consul.hashicorp.com/connect-service - For pods that accept inbound connections, this specifies the name of the service that is being served. This defaults to the name of the first container in the pod.

If using ACLs, this must be the same name as the Pod's ServiceAccount.

https://www.consul.io/docs/k8s/connect#available-annotations

Unrelated to Consul Connect, if I'm using the Consul Kubernetes Auth method, then I also need to make a connection between a service account and a consul service. But here I do have the ability to override the default name (which is the name of the service account) by using an annotation on the service account:

The Consul leader makes an additional query, this time to the ServiceAccount API to check for the existence of an annotation of consul.hashicorp.com/service-name on the ServiceAccount object. If one is found its value will override the trusted attribute of serviceaccount.name for the purposes of evaluating any binding rules.

https://www.consul.io/docs/security/acl/auth-methods/kubernetes#kubernetes-authentication-details

Would it make sense for consul connect to recognize the same annotation in order to make those two kubernetes/consul integrations match more nicely?

[lkysow]

Yeah it would make sense to support the same annotation on both. We have some upcoming changes around service naming so I think we'll delay on this until then though.

[whiskeysierra]

Now I'm curious. Can you share something about that already?

[lkysow]

We're looking at tying in Service objects and relying less on pod annotations. See https://github.com/hashicorp/consul-k8s/pull/445 for a very early draft.

In the end you'll still need a service account (if ACLs are enabled) so the annotation matching would still probably make sense.

[whiskeysierra]

We're looking at tying in Service objects and relying less on pod annotations.

Are you referring to standard Kubernetes Service resources or is that going to be a Consul CRD?

[lkysow]

Standard kube Service objects.

[whiskeysierra]

Is this related to the move from binding envoy locally to running it in transparent mode? I'm a bit confused because the Service resource type is an odd choice otherwise. As of right now, anyone who is using Consul Connect has effectively almost 0 use cases for Service resources - since they are not used for service discovery at all.

(The only use cases that I can see are headless services for stateful sets and LoadBalancer services for ingress.)

[lkysow]

Is this related to the move from binding envoy locally to running it in transparent mode?

Yes exactly. When we support using Kube DNS transparently then we need to know about Service objects since that's the backing of Kube DNS. You're correct that users of Consul Connect don't need services right now but we know that most users don't start their deploys into k8s directly with Consul Connect and instead migrate to it after they decide they need the feature set.

[hamishforbes]

Is there any update on this issue, supporting overriding the service name somehow.

I've run into a bit of a show stopper trying to put some rudimentary namespacing into my service / intention setup, essentially my k8s configuration automation is prefixing Connect service names with the k8s namespace and generating the correct ServiceIntention CRDs to match.

My Consul servers exist outside of k8s so I'm not using all the magic ACL configurations. I changed my consul auth method to prefix the bind-name with the k8s namespace and I think that should be enough for my setup to work. Except the webhook is now rejecting the pods as the serviceaccount name doesnt match the service name, even though the auth method should generate a valid ACL.

I could just have my automation prefix the serviceaccount name as well but... this then breaks my Vault integration, because that is tied to specific service account names as well, but with the namespace configured separately.

At the moment I'm going to have to manually (e.g. outside of k8s) create either a ton of intentions or a ton of Vault k8s roles and I think the only thing blocking my ideal solution working is the admission webhook incorrectly rejecting the pod

edit: Looking at the codebase, has this check been removed in 0.26.0 anyway?

[kschoche]

Hi @hamishforbes !

Is there any update on this issue, supporting overriding the service name somehow.

Since this issue was opened we've done a pretty large refactor of the way mutation and registration is handled, it is part of the larger transparent proxy changes which Luke mentioned above. Unfortunately that refactor alone isn't enough to support overriding the service name on a serviceaccount. We did a bit of investigation internally and found that due to the way certificates are signed by Consul we'd need to also have the webhook update ACL policies and that would require adding acl: write to the webhook, which creates a very large security risk should there be an RCE attack.

edit: Looking at the codebase, has this check been removed in 0.26.0 anyway?

We still have the requirement that serviceAccountName is the same as the consul service name, it's been moved to the init container inside of connect-init.

[david-yu]

Hi all just wanted to provide an update. There currently isn't a plan to support utilizing arbitrary service accounts for services at the moment within Consul K8s due to the security concerns mentioned previously.

We do have solutions built into our enterprise products that will solve these organizational problems at scale that will help with issues where collisions with service names and service account names happen at scale. Although it is definitely still possible to manage this through user-provisioned automation that you would implement and manage on your own.

Admin Partitions which was released in Consul 1.11, will solve the conflicts at a cluster level, where where you have multiple teams that are autonomous (and not coordinating names and services accounts) and deploying across multiple K8 clusters within a single Consul DC (and in the future multiple Consul DCs). Namespaces also currently help with mirroring K8s namespaces in Consul and prevent conflicts at a namespace level.

[david-yu]

Closing as there is no intention to support arbitrary service names.