Publish IPs from ClusterIP and NodePort services on Ingresses

[maxnitze]

Do you want to request a feature or report a bug?

Feature

What did you expect to see?

I'm trying to setup ArgoCD, which is using the status of an Ingress for its health checks.

In the Traefik config there is the option to publish the service status into all Ingresses by adding the following to the static config:

providers:
  kubernetesIngress:
    ingressEndpoint:
      publishedService: <NAMESPACE>/<TRAEFIK_SERVICE_NAME>

This is only doing anything for LoadBalancer type services though, as the status is not set on other types of services.

When not running in the cloud it is pretty common, that the traefik service is not of type LoadBalancer, but a ClusterIP service with external IPs configured or even a NodePort service.

A working solution is, to configure one of the the following instead:

ClusterIP service with external IP 80.70.60.50

providers:
  kubernetesIngress:
    ingressEndpoint:
      ip: 80.70.60.50

NodePort service with one node having IP 10.20.30.40 and hostname node1.my-domain.com

providers:
  kubernetesIngress:
    ingressEndpoint:
      ip: 10.20.30.40

or

providers:
  kubernetesIngress:
    ingressEndpoint:
      hostname: node1.my-domain.com

There are multiple problems with this solution:

• It seems that the configured IP and/or hostname is completely arbitrary. So you can configure anything (that is a valid hostname or valid IP) and it does not need to have any connection to your cluster whatsoever.
• In case of the ClusterIP service you have to duplicate the IP and manually keep both places in sync to keep it consistent.
• You cannot reflect, that a ClusterIP service has multiple external IPs or a NodePort is open on multiple nodes.

Proposal

In case the service, that is configured on the publishedService field, is a ClusterIP service all IPs given in the list of externaIPs are copied to the Ingress status.

In case the service, that is configured on the publishedService field, is a NodePort service all IPs of all nodes are set in the Ingress status.

Links

The original feature request: https://github.com/traefik/traefik/issues/2173 The question I asked on the community forums: https://community.traefik.io/t/usage-of-publishedservice-in-externalip-setup/9971

[maxnitze]

Unfortunately I have no experience in Go yet, but I would gladly help implementing this feature.

I already had a look at the code and I think I found the place to adapt: https://github.com/traefik/traefik/blob/v2.4/pkg/provider/kubernetes/ingress/kubernetes.go#L315

And here's an idea on how to do this:

func (p *Provider) updateIngressStatus(ing *networkingv1beta1.Ingress, k8sClient Client) error {
	[ ... ]

	service, exists, err := k8sClient.GetService(serviceNamespace, serviceName)
	if err != nil {
		return fmt.Errorf("cannot get service %s, received error: %w", p.IngressEndpoint.PublishedService, err)
	}

	if !exists {
		return fmt.Errorf("missing service: %s", p.IngressEndpoint.PublishedService)
	}

	if service.Spec.Type == corev1.ServiceTypeClusterIP {
		loadBalancerIngresses := make([]corev1.LoadBalancerIngress, len(service.Spec.ExternalIPs))
		for i, ip := range service.Spec.ExternalIPs {
			loadBalancerIngresses[i] = corev1.LoadBalancerIngress{ IP: ip }
		}
		return k8sClient.UpdateIngressStatus(ing, loadBalancerIngresses)
	} else if service.Spec.Type == corev1.ServiceTypeNodePort {
		// TODO this is only an idea to get all nodes and their (internal) IPs
		// maybe this also misses a filter for all nodes with taint `node-role.kubernetes.io/master: NoSchedule`
		// or maybe something like all workers only (all without label `node-role.kubernetes.io/master`)
		nodes := k8sClient.GetNodes() // this does not exist
		loadBalancerIngresses := make([]corev1.LoadBalancerIngress, len(nodes))
		for i, node := range nodes {
			loadBalancerIngresses[i] = corev1.LoadBalancerIngress{ IP: node.InternalIP }
		}
		return k8sClient.UpdateIngressStatus(ing, loadBalancerIngresses)
	} else if service.Status.LoadBalancer.Ingress != nil {
		return k8sClient.UpdateIngressStatus(ing, service.Status.LoadBalancer.Ingress)
	} else {
		// service exists, but has no Load Balancer status
		log.Debugf("Skipping updating Ingress %s/%s due to service %s having no status set", ing.Namespace, ing.Name, p.IngressEndpoint.PublishedService)
		return nil
	}
}

I could not find any feature to get all nodes (yet). But I tried to roughly describe my idea there.

What do you think? Would this be a viable solution?

Is this feature a good idea in general? Or is this somehow misusing the field for something it is not designed for?

[davehamer]

Hi All, I've now run into this issue on my own cluster. I've tried the workarounds but they don't seem to work at the moment - the status does not get updated on the svc. (and then I get the message: Skipping updating Ingress xxx due to service kube-system/traefik having no status set Has something changed again since this issue was written? I was going to write on the community forums, however the original thread was locked and can no longer be updated... Kind Regards; Dave

[fdelucchijr]

What do you think? Would this be a viable solution?

Is this feature a good idea in general? Or is this somehow misusing the field for something it is not designed for?

I think this is a very good idea. In my case, the benefit is to use the K8s as a Configuration Discovery Provider for an external Traefik host. This allows me to centralize all the hosts in my network into one instance of Traefik with all the auto-configuration utilities it has. The draft code you add seems good and explains well your idea.

Can we have a confirmation of a member or a contributor this is in the Traefik Roadmap? Or should this be added as a plugin?

EDIT: I've seen this issue has already been marked as in the v2 roadmap, probably best to start. ~I will soon add a PR for the Feature.~

EDIT2: This end up getting out of my radar.I Found an alternative solution for using K8s with external traefik.

[peterhoneder]

whenever you try setting the ip setting for the ingressEndpoint, make sure you are not setting publishedService, because it will override the IP setting and then it won't work

[ddtmachado]

Hey, we think this proposal might make sense but unfortunately, this would not make it to our roadmap for a while as we are focused elsewhere. If you or another community member would like to build it, let us know and we will work with you to make sure you have all the information needed.

We prefer to work with our community members at the beginning of the design process to make sure we are aligned and can move quickly with the review and merge process.

If you feel the current design is ready for a PR feel free to open and we can continue iterating from there just don’t forget to check out the contributor docs and to link the PR to this issue.

[sspreitzer]

@ddtmachado "this would not make it to our roadmap for a while as we are focused elsewhere.", makes me refrain from advocating Traefik Enterprise in our company and to others.

[wrenix]

For support multiple Addresses (e.g. ipv4 and ipv6), ingressEndpoint should be an array:

providers:
  kubernetesIngress:
    ingressEndpoint:
      - ip: 10.20.30.40
      - ip: 2a01:4f8::1

[tgajdo]

Still no solution for this? I'm banging my head to solve it somehow...

[maxnitze]

There are solutions (rather Workarounds?) for ClusterIP and NodePort services posted in my initial question.

But afaik there is no built-in feature to automatically propagate those to Ingresses, no.

[tgajdo]

@maxnitze I was able to use your workaround, but still it's a bit odd that after 3 years this is still a thing...

[maxnitze]

I guess most people either use a public cloud provider or on-premise clusters with something like MetalLB, so having non-LoadBalancer services for your ingress controller is not really a thing anymore.

I am also not using any such service anymore. So this is not an issue for me anymore.