Support for using service upstream instead of ingress endpoints

[pedroosorio]

trafficstars

To leverage the cluster's internal load balancing (and other features) we should allow the usage of the service upstream (like ingress-nginx allows) to be used instead of the service's endpoints.

Add a "nginx.org/service-upstream: 'true'" annotation support to that, in that case, the upstream is the service ip/port itself and not the endpoints.

Thank you.

[github-actions[bot]]

Hi @pedroosorio thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this :slightly_smiling_face:

Cheers!

[brianehlert]

The service endpoint does not provide 'loadbalancing' functionality (it is neither equivalent to 'random' or 'roundrobin' as we think of it in the networking space). Where using the dataplane to balance directly to the pods provides greater control of traffic steering, cookie support, sticky sessions, etc.

Can you please help me understand what benefit you feel this brings to you?

[jasonwilliams14]

@pedroosorio We have this capability in our CRD resources (virtualServer/router) but not Ingress resource.

use-cluster-ip.

https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#common-parts-of-the-virtualserver-and-virtualserverroute

Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like lb-method and next-upstream) will have no effect, as the Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP.

[pedroosorio]

@brianehlert I understand the value of using the pods directly, it is optimal in most cases. In my case, I have a service with an endpoint pointing to an external IP (my local machine, for instance). The ingress correctly detects the endpoints, but not the ingress (upstreams are 127.0.0.1:8181). I would want this feature to work around this issue actually. Tho, I think it would be an option one could take in other scenarios as well, losing the capabilities you mentioned.

[brianehlert]

Using an Ingress Controller to proxy / redirect to an external service is not unusual. Thanks for describing that. Let me see if we can get an example posted in the thread.

[pedroosorio]

@brianehlert my goal is to have the original ingress and service(s) of an application and running the application on the host machine. That works fine in terms of traffic (ingress controller can reach the app in the localhost). To do that, I created a custom endpoint for that service which seems to work just fine if I talk to the service cluster IP. The nginx ingress controller is not picking the endpoint IP tho. Let me know if there is a better way of accomplishing this and thank you !

[pedroosorio]

@brianehlert I worked around the issue by having an annotation for the service. Instead of proxy_pass to the upstream, in those scenarios, I proxy_pass to the service. I've created the services in port 80 to facilitate the code.

I've opened another issue with this project (tho I don't seem to find it), which is an extremely limited set of functions in the template language. We have almost no functions to play with, which we require for several workarounds. Should I open that request separately? That ties well into working around issues like this.

[wc-s]

Another use case is that the linkerd service mesh recommends using service upstream if we want to hook the ingress controller up to the mesh: https://linkerd.io/2.12/tasks/using-ingress/#nginx

Although in this case they have an "ingress mode" to work around this: https://linkerd.io/2.12/reference/proxy-configuration/#ingress-mode

Basically linkerd wants to take over doing the load balancing, which as a service mesh user is what I want too.

[brianehlert]

This project supports targeting service endpoint instead of pods. Allowing the load balancing to shift to a downstream mesh instead of ingress performing the loadbalancing.

This was first implemented to support an Istio backend with v1.11.

The setting is use-cluster-ip described here: https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#upstream

We recommend you use the project CRDs to gain this and other improved capabilities.

[wc-s]

Thanks for the reply. I've seen those and yea we don't use the CRDs. I'll figure out what's the best way forward for us.

[brianehlert]

@wc-s Might you help me understand why you choose not to use the CRDs?

[wc-s]

Mainly so that we do not have vendor lock-in, and secondly because knowledge of Ingress exists in the organization, whereas the CRDs take some additional learning.

It's a bit subjective, especially given how the Ingress annotations are already vendor-specific, but that's where my company and my own sentiment lean towards.