__meta_kubernetes_service_loadbalancer_ip not working as expected

[whitepiratebaku]

trafficstars

What did you do?

used following config snippet to set LB service external IP as param_target for DNS probes

....
      relabel_configs:
      - action: keep
        regex: powerdns-external;pdns-udp
        source_labels:
        - __meta_kubernetes_service_name
        - __meta_kubernetes_service_port_name
      - source_labels:
        - __meta_kubernetes_service_loadbalancer_ip
        target_label: __param_target
...

now with following config __param_target not set, it is empty. If I check powerdns-external service I see that it has loadBalancer IP only under status.loadBalancer.ingress so no .spec.loadBalancerIP defined automatically, looks like after 1.24 Kubernetes dropped this .spec.loadBalancerIP setting.

apiVersion: v1
kind: Service
metadata:
  name: powerdns-external
.....
status:
  loadBalancer:
    ingress:
      - ip: 192.168.1.15
spec:
  ports:
    - name: pdns-udp
      protocol: UDP
      port: 53
      targetPort: 53
      nodePort: 30937
  selector:
    application: powerdns
  clusterIP: 10.96.26.217
  clusterIPs:
    - 10.96.26.217
  type: LoadBalancer
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster

Main point if I set .spec.loadBalancerIP manually from service then scrape job starts to work and thus __meta_kubernetes_service_loadbalancer_ip returns loadBalancerIP value. So could it be that __meta_kubernetes_service_loadbalancer_ip relies on .spec.loadBalancerIP and not compatible with latest Kubernetes?

What did you expect to see?

No response

What did you see instead? Under which circumstances?

__meta_kubernetes_service_loadbalancer_ip returned no value

System information

No response

Prometheus version

No response

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[machine424]

Thanks for reporting this.

I assume that even when .spec.loadBalancerIP was the only way to configure an LB, .status.loadBalancer was reflecting that and should have been used.

So, yes, I think we should switch to .status.loadBalancer (assuming the above assumption is correct) to support the new "configuration via annotation" method as well.

We should also consider which __meta_xxx(.status.loadBalancer is an array) to use. Perhaps we should expose LoadBalancerIngress.Hostname when available...

[jeffreylean]

Hi @machine424 I would like to work on this. I just need more clarification on the LoadBalancerIngress.Hostname part. When you say expose, we will have to populate the LoadBalancerIngress.Hostname to one of the __meta_kubernetes field if the hostname is available and there are multiple IP in the .status.loadBalancer array ?

[Garvit-77]

@jeffreylean Are you still working on this issue ? if not I would like to give it a try

[machine424]

Hi @machine424 I would like to work on this. I just need more clarification on the LoadBalancerIngress.Hostname part. When you say expose, we will have to populate the LoadBalancerIngress.Hostname to one of the __meta_kubernetes field if the hostname is available and there are multiple IP in the .status.loadBalancer array ?

Yep, IIUC, sometimes, only a hostname is available and we may need to expose it in a separate label. But let's stick to the IPs for now.

[hansbogert]

We should also consider which __meta_xxx(.status.loadBalancer is an array) to use. Perhaps we should expose LoadBalancerIngress.Hostname when available...

I'm interested in this. Perhaps they should become separate discoveries like is already done for the ports array in a svc/loadbalancer resource.

If we don't make them separate, and go for something like this:

__meta_xxx.. = "1.1.1.1,2.2.2.2"

it'll be hard to scrape with a prometheus job, it will likely result in all kinds of regexp matches and possibly needing multiple scrape jobs for each position in the comma separated string. At least I'm running into this with the vendor specific annotations which often have multiple IPs encoded as a comma separated string.

[machine424]

We should also consider which __meta_xxx(.status.loadBalancer is an array) to use. Perhaps we should expose LoadBalancerIngress.Hostname when available...

I'm interested in this. Perhaps they should become separate discoveries like is already done for the ports array in a svc/loadbalancer resource.

If we don't make them separate, and go for something like this:

__meta_xxx.. = "1.1.1.1,2.2.2.2"

it'll be hard to scrape with a prometheus job, it will likely result in all kinds of regexp matches and possibly needing multiple scrape jobs for each position in the comma separated string. At least I'm running into this with the vendor specific annotations which often have multiple IPs encoded as a comma separated string.

It's a tradeoff between defining extra metadata that will only be relevant and make sense to a small portion of users, and exposing as minimal metadata as possible, in raw format, leaving it to those users to extract the relevant data themselves.

If the order in the array doesn't change, adding metadata containing a comma-separated list of IPs/hostnames wouldn't be a first; we already do this in other service discoveries, and relabeling seems to be sufficient to extract the relevant info.

I'd like to know more about how your Spec.LoadBalancer look. If you could share more about that, it will help us determine the best approach.

[hansbogert]

I'd like to know more about how your Spec.LoadBalancer look. If you could share more about that, it will help us determine the best approach.

I'm not sure what 'my' spec.LoadBalancer is in this case?

[machine424]

The definition/manifest of the Service

[tiesmaster]

@machine424 The Service can have various shapes, but all of them will have the configuration in the annotations, as that's the way forward atm, with the .spec.LoadBalancerIP field being deprecated. These are all different, depending on the vendor, and maybe report differently in the .status field.

Here are a couple examples of services that we have in our clusters (also holds for @hansbogert, we're colleague's), with also how this ends up in the .status field. This is not an exhaustive list, but I hope this gets enough data points across to move forwards. Otherwise, I can provide more.

Metallb (ipv4 only)

apiVersion: v1
kind: Service
metadata:
  annotations:
    ...
    metallb.universe.tf/ip-allocated-from-pool: public
    metallb.universe.tf/loadBalancerIPs: '1.2.3.4'
    ...
spec:
  type: LoadBalancer
  # loadBalancerIP is absent here
  ...
status:
  loadBalancer:
    ingress:
    - ip: 1.2.3.4
      ipMode: VIP

Metallb (dual stack)

apiVersion: v1
kind: Service
metadata:
  annotations:
    ...
    metallb.universe.tf/ip-allocated-from-pool: public
    metallb.universe.tf/loadBalancerIPs: '1.2.3.4,2001:1:2:3:4::'
    ...
spec:
  type: LoadBalancer
  # loadBalancerIP is absent here
  ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: PreferDualStack
  ...
status:
  loadBalancer:
    ingress:
    - ip: '2001:1:2:3:4::'
      ipMode: VIP
    - ip: 1.2.3.4
      ipMode: VIP

AKS

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: rg-...
    service.beta.kubernetes.io/azure-pip-name: pip-...
spec:
  type: LoadBalancer
  # loadBalancerIP is absent here
  ...
status:
  loadBalancer:
    ingress:
    - ip: 1.2.3.4
      ipMode: VIP
    # If the cluster is dual-stack enabled, then the ipv6 IP is also reported back here

EKS EKS doesn't report back any IPs, but rather works on the level of a DNS entry that points to 3 IPs.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-name: example-lb-name
    ...
spec:
  type: LoadBalancer
  # loadBalancerIP is absent here
  ...
status:
  loadBalancer:
    ingress:
    - hostname: example-lb-name-ID_OF_THE_LB.elb.eu-central-1.amazonaws.com

As you can see, most of the vendors report the IPs back in a standardized way in the .status.loadBalancer.ingress array. AWS is a bit different, as that doesn't provide an hostname, but it still follows the same convention.

I hope this helps to determine the best approach here. Let me know if you need more input.