Support addressing individual pods in a headless service

[derekperkins]

Particularly when using StatefulSets, it can be valuable to expose individual pods through an ingress. This feature request was well phrased in this StackOverflow question: https://stackoverflow.com/questions/46456239/how-to-expose-a-headless-service-for-a-statefulset-externally-in-kubernetes

The goal is to expose the individual Kafka Brokers externally which are internally addressed as:

kafka-0.broker.kafka.svc.cluster.local:9092
kafka-1.broker.kafka.svc.cluster.local:9092 
kafka-2.broker.kafka.svc.cluster.local:9092

The constraint is that this external service be able to address the brokers specifically. Whats the right (or one possible) way of going about this? Is it possible to expose a external service per kafka-x.broker.kafka.svc.cluster.local:9092?

Here is how an HAProxy ingress controller handles it with their custom ingress CRD https://appscode.com/products/voyager/8.0.1/guides/ingress/http/statefulset-pod/#forward-traffic-to-specific-pods-of-a-statefulset

Here's a way to do this without an ingress controller, but requires a LoadBalancer per pod https://itnext.io/exposing-statefulsets-in-kubernetes-698730fb92a1

[davecheney]

Thanks for raising this issue.

When you say headless, do you mean, doesn't have a service object? If so, why not add a service object?

[derekperkins]

In this instance, I mean clusterIP: None in a service that is attached to a StatefulSet, which makes each pod addressable via an internal dns entry, like the kafka entries above.

pod-ordinal.service.namespace.svc.cluster.local

The key point is that I don't want the service to randomly assign my request to one of the underlying pods like a traditional service. I would like to set up kafka-0.myapp.dev or kafka.myapp.dev/kafka-0 in public dns to route through the ingress directly to kafka-0.broker.kafka.svc.cluster.local.

[davecheney]

Thanks for your reply. Just to check I'm understanding your request, you want the ability to direct ingress traffic directly to one pod?

[derekperkins]

Correct. I would like to be able to direct ingress traffic to any specific pod matched by the service selector, whether that is only 1 pod or 1 of 50 pods. The choice of which pod to redirect to could be present either as a subdomain kafka-0.myapp.dev or as a path prefix kafka.myapp.dev/kafka-0

[davecheney]

What's the use case for this? Is it that you want to be able to route to a specific pod because you need something like sticky sessions or header based routing?

[derekperkins]

I am running ~100 MySQL instances inside of k8s in StatefulSets using Vitess. Each pod in the statefulset exposes a number of metric and debug web pages. It is very cumbersome to always have to run kubectl port-forward to check on things manually Additionally, if you are scraping metrics from outside the cluster, you need reliable paths to collect the metrics per pod. It also requires any DBAs or consultants to have k8s experience and kubectl credentials when they likely shouldn't need them.

Alternatives I have investigated require creating a service per pod, which seems like a hacked solution to the problem. That only works for StatefulSets, and only if those StatefulSets don't scale without simultaneously increasing/decreasing the service count. Using LoadBalancers for each of those services increases cost and complexity even more.

Going through the ingress controller gives you access to all the underlying pods as they scale up and down. You also get the benefit of security and auth, so all the same ingress logic extends to these pods.

Using a headless service puts all the appropriate internal dns entries in place, so it seems like it wouldn't be too hard to just map a path prefix to the service subdomain and return a 404 if that subdomain doesn't exist. That may not be the best way to implement it, I'm not an expert, but seems like a good place to start. I use the same principle for all my intra-cluster communication, so it feels like the same mechanism deserves a place in cross-cluster communication.

[stevesloka]

Hey @derekperkins, we have support for external named services now. Could you use the dns name of the pod to set up a route to the specific instance you are looking to connect to?

[moderation]

I'm still trying to get my head around clusterIP: None vs. ExternalName and I think the difference is that ExternalName creates CNAME DNS records where clusterIP: None uses A records. What I was wondering is if it is possible to remove the first Envoy proxy pod in the diagram at https://blog.markvincze.com/how-to-use-envoy-as-a-load-balancer-in-kubernetes/#4-create-the-envoy-deployment by having Contour route directly to the headless service endpoints (bypassing kube-proxy).

Related to this, NGINX and Zalando's Skipper both say they route to endpoints and not services. I believe Contour is doing the same with EDS as outlined in the external names documentation.

[davecheney]

Contour does not use kube proxy, we configure Envoy to communicate directly to the addresses listed in the Endpoint document.

On Tue, 23 Jul 2019 at 10:56, moderation [email protected] wrote:

I'm still trying to get my head around clusterIP: None vs. ExternalName and I think the difference is that ExternalName creates CNAME DNS records where clusterIP: None uses A records. What I was wondering is if it is possible to remove the first Envoy proxy pod in the diagram at https://blog.markvincze.com/how-to-use-envoy-as-a-load-balancer-in-kubernetes/#4-create-the-envoy-deployment. Create a headless service and have Contour route directly to the endpoints (bypassing kube-proxy).

Related to this, NGINX https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#why-endpoints-and-not-services and Zalando's Skipper https://opensource.zalando.com/skipper/kubernetes/ingress-controller/#why-skipper-uses-endpoints-and-not-services both say they route to endpoints and not services. I believe Contour is doing the same with EDS as outlined at https://github.com/heptio/contour/blob/master/design/external-names.md#background

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/heptio/contour/issues/957?email_source=notifications&email_token=AAABYA2FQ4AQVMOW5F4JYC3QA5A3LA5CNFSM4HAPUWJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2T5W2Q#issuecomment-514317162, or mute the thread https://github.com/notifications/unsubscribe-auth/AAABYA6ZAZRX3CIJ3SDXBLLQA5A3LANCNFSM4HAPUWJQ .

[moderation]

That confirms my assumption which is great news. Might be worthwhile calling this out more explicitly in the documentation. I might take a stab at a doco PR if this make sense.

[DazWilkin]

I'm interested in the solution to help port an HPC-like platform to k8s. I'm unfamiliar with Contour and @moderation's epiphany is not immediately obvious to me. Would appreciate a brief synopsis of how do this. Thanks!

[davecheney]

Daz,

Let’s not mix the streams. Please open a new feature request and outline the problem that you have that you’d like to solve with contour.

Thanks

Dave

On 15 Nov 2019, at 03:08, Daz Wilkin [email protected] wrote:

 I'm interested in the solution to help port an HPC-like platform to k8s. I'm unfamiliar with Contour and @moderation's epiphany is not immediately obvious to me. Would appreciate a brief synopsis of how do this. Thanks!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[DazWilkin]

Apologies @davecheney, this issue describes my problem.

I'm after the "Might be worthwhile calling this out more explicitly in the" documentation if it exists.

IIUC @moderation appears to believe that it's possible to externally access individual StatefulSet Pods.

[michmike]

@derekperkins would you be able to attend one of the contour community meetings? we can discuss this use case there.

[derekperkins]

Sure, when are those?

[davecheney]

There is one tomorrow at 15:30 pacific

https://projectcontour.io/community/

[nefelim4ag]

I'd suggest a simple workaround, I will not suggest a proper configuration for HTTPProxy. Kubernetes define additional labels on STS pods:

statefulset.kubernetes.io/pod-name: {{ $.Release.Name }}-{{ $index }}

https://github.com/Nefelim4ag/teamcity-charts/blob/main/charts/teamcity-server/templates/teamcity/service.yaml.tpl#L51 So, you can define service per Pod, and expose STS as several individual services.

The tricky part is that you don't have instruments to route by hostname with wildcard domain, but you still can generate several services and several ingress/httpProxy objects for that.