kubernetes-sd: expose status.PodIPs / podMonitor support IP family

[akunszt]

Proposal

It would be great if the kubernetes SD can expose the status.podIPs field as well as the status.podIP.

It is quite useful in dual-stack environments where the status.podIP (the first/primary IP address of the pod) has to remain IPv4 to avoid any issues with readiness and liveness checks but the monitoring for some pods using a PodMonitor already can be migrated to IPv6. Now the PodMonitor picks the status.podIP and there is no way to change that so we will learn if the monitoring works on IPv6 when we switch to IPv6 primary in the whole cluster.

The best would be if this can be configured in the podMonitor like ipFamilies in the Service or something but only if the label is exposed then a relabel configuration can be used to change the __address__ which is also sufficient.

[DrAuYueng]

/assign

[machine424]

We can also use the progress made on this issue to check if the pod and endpoints roles pick the same __address__ for dual-stack Pods. Both roles should select either the IPv4 or the IPv6 address, for consistency.

i.e. let’s verify whether the result from https://github.com/prometheus/prometheus/blob/2e58d46522c328124a0eefd27887b784c2a9413d/discovery/kubernetes/endpoints.go#L316-L320 matches the value in .status.podIP used by the pod role. If not, we’ll need to document this difference.

[DrAuYueng]

@akunszt Like this?

__meta_kubernetes_namespace="default"
__meta_kubernetes_pod_container_init="false"
__meta_kubernetes_pod_container_name="nginx"
__meta_kubernetes_pod_container_port_number="80"
__meta_kubernetes_pod_container_port_protocol="TCP"
__meta_kubernetes_pod_controller_kind="ReplicaSet"
__meta_kubernetes_pod_controller_name="nginx-dual-stack-69b9449db"
__meta_kubernetes_pod_host_ip="192.168.1.59"
__meta_kubernetes_pod_ip="100.117.147.148"
__meta_kubernetes_pod_ips="100.117.147.148,2001::3238:634c:2ed4:ed00"

or

__address__="100.117.147.148:80"
__meta_kubernetes_namespace="default"
__meta_kubernetes_pod_container_init="false"
__meta_kubernetes_pod_container_name="nginx"
__meta_kubernetes_pod_container_port_number="80"
__meta_kubernetes_pod_container_port_protocol="TCP"
__meta_kubernetes_pod_controller_kind="ReplicaSet"
__meta_kubernetes_pod_controller_name="nginx-dual-stack-69b9449db"
__meta_kubernetes_pod_host_ip="192.168.1.59"
__meta_kubernetes_pod_ip="100.117.147.148"
__meta_kubernetes_pod_ipv6="2001::3238:634c:2ed4:ed00"

[akunszt]

Thanks. Currently both solutions would do for us. I lean a little bit to the second one because when we change the IP ordering and make IPv6 the default then we don't have to modify the PodMonitors. Also please note that a pod can have multiple IPv4 and IPv6 addresses and even network interfaces but I think giving back the first IPv4 and IPv6 address in an easy to use label is a good starting point and likely this will be sufficient for everyone.

[DrAuYueng]

I lean a little bit to the second one because when we change the IP ordering and make IPv6 the default then we don't have to modify the PodMonitors.

Can I take a look at the PodMonitor configuration?

[DrAuYueng]

@akunszt How do you change the IP sorting? In dual-stack scenarios,

1. CNI plugin, such as calico, always use IPv4 as first IP.
2. CRI use first IP as the primary IP, the rest as additional IPs.
3. Kubelet use primary IP as podIP, podIPs include all IPs, eg: [100.117.147.148, 2001::3238:634c:2ed4:ed00]

[akunszt]

@DrAuYueng

1. We are using a patched aws-vpc-cni where we can control of the ordering but it is actually up to the CNI if the pod is on the pod network. If it is a host networking pod then it will use the node's IP ordering which can be configured on the kubelet. After we are ready we will change the IP ordering both in the CNI and in the kubelet from IPv4,IPv6 to IPv6,IPv4 and then later - much later - to IPv6 only. At last that's the notion.
2. Yes, that's correct. We can't change that to IPv6 as every readiness and liveness check will start using that and we have a few IPv4-only deployments.
3. Yep, also correct.

Now we have this podMonitor configuration (just the spec):

spec:
  jobLabel: app
  namespaceSelector:
    any: true
  podMetricsEndpoints:
    - port: metrics
  podTargetLabels:
    - mirrormaker
  selector:
    matchLabels:
      app: mirrormaker

With your changes we can do something similar (I write this without testing):

spec:
  jobLabel: app
  namespaceSelector:
    any: true
  podMetricsEndpoints:
    - relabelings:
        - action: replace
          sourceLabels: [ __meta_kubernetes_pod_ipv6 ]
          targetLabels: "__address__"
          replacement: "[$1]:1234"
  podTargetLabels:
    - mirrormaker
  selector:
    matchLabels:
      app: mirrormake

We might need to add a regex to the relabelings if we have __meta_kubernetes_pod_ips like:

regex: ",.+?,(.+?),"

As we have only one PodMonitor I strongly hope that this - the relabeling - works.

[bboreham]

Hello from the bug-scrub! @DrAuYueng are you still interested in this feature?

[DrAuYueng]

@bboreham I will continue to work on this issue.

[DrAuYueng]

I've submitted a PR to fix this issue. cc @simonpasquier @machine424 @bboreham