Random IP address from the set of resolved IP addresses

[mabrarov]

Problem:

I use OpenShift 3.11 with headless OpenShift (K8s) service. The pods, which I run in OpenShift, use TLS certificate issued by OpenShift CA. I want to use Blackbox exporter to monitor (using Prometheus rules and Alertmanager) expiration date of TLS certificates used by each pod to which the OpenShift service points. Due to OpenShift service is headless, when the name of the service (used in Blackbox exporter target) is resolved, then multiple IP addresses are returned - IP addresses of all pods to which the service points. The set and the order of resolved IP addresses remains the same b/w probes (it seems that K8s - I've tried a single node K8s with Minikube - uses round-robin DNS for headless K8s service).

Blackbox exporter 0.22.0 tries just the first IP address when the target hostname is resolved into multiple IP addresses. This prevents Blackbox exporter probes reaching all pods behind headless OpenShift service.

Solution:

Introduce random_resolved_ip probe option with default value false (i.e. by default just the first IP address is used, which matches behavior of Blackbox exporter 0.22.0). When random_resolved_ip is true, then Blackbox exporter prober uses random address from the list of resolved IP addresses. This behavior leads to all pods behind a headleess OpenShift service are reached by the probe eventually (which is considered acceptable tradeoff for my use case, due to the frequency of probes covers lifetime of TLS certificate issued by OpenShift CA).

Changes:

Using random IP address from the list of resolved IP addresses when requested by random_resolved_ip probe option (the first IP address is used otherwise, which is backward compatible). This ensures that all target hosts are hit by the probes eventually.

Notes:

Refer to mabrarov/haproxy-test/pull/1 for the manual testing of this pull request with:

1. Docker (using Docker Compose and Docker DNS, tested on CentOS 7.9). This case just confirms that Docker DNS uses round-robin DNS, which resolves the problem (i.e. without changes in Blackbox exporter 0.22.0).
2. K8s (setup with Minikube, tested on Ubuntu 18.04). This case just confirms that K8s uses round-robin DNS for headless service, which resolves the problem (i.e. without changes in Blackbox exporter 0.22.0).
3. OpenShift Origin 3.11 (setup with oc cluster up command, tested on CentOS 7.9). This case makes it possible to reproduce the problem (using http module) as well as to validate suggested solution (using http_random_ip module).

[SuperQ]

DNS ordering sounds like an RFC 3848 Section 6 Rule 9 issue. Depending on what OS version you're on, you can fix this with changes to /etc/gai.conf. IIRC, Redhat has some specific docs and patches to this.

[mabrarov]

Hi @SuperQ,

Depending on what OS version you're on, you can fix this with changes to /etc/gai.conf

Sorry, but I failed to find information about implementation of round-robin DNS by the means of /etc/gai.conf, i.e. it doesn't seem that /etc/gai.conf can be used to implement (not to turn off sorting) round-robin DNS. I believe the sorting of IP addresses comes from OpenShift in my case and /etc/gai.conf cannot be used randomize the list of IP addresses.

OpenShift Container Platform 3.11. Architecture. Networking:

This DNS structure also covers headless services, where a portal IP is not assigned to the service and the kube-proxy does not load-balance or provide routing for its endpoints. Service DNS can still be used and responds with multiple A records, one for each pod of the service, allowing the client to round-robin between each pod.

I guess, it means that OpenShift 3.11 DNS relies on client to implement round-robin.

[SuperQ]

Sorry, it's been a while since I looked into the gai.conf specifics. I know there are some issues with RedHat glibc that cause the sorting. At a previous job, we manually patched Debian glibc to fix the sorting issue.

In the case of headless services, you're supposed to monitor all endpoints, not pick a random one.

[mabrarov]

Hi @SuperQ,

In the case of headless services, you're supposed to monitor all endpoints, not pick a random one.

For the case I have (monitoring of TLS certificate expiration date), random endpoint is OK, because certificate is issued by OpenShift CA for 1-2 years and alert starts triggering if there are less than 30 days till expiration date. It means that random IP most probably will hit all pods using old TLS certificate if probe frequency is 30-150 seconds. It's just monitoring - not the complete solution (which also includes failing liveness probe if expired TLS certificate is used).

I understand that randomizing target IP address is not ideal solution, but in case its implementation is simple (trivial?) and configurable (default behavior remains the same), then why not support it in Blackbox exporter?

[mabrarov]

Hi @SuperQ, @roidelapluie and @electron0zero,

Is there something I need to do (i.e. is there something pending from my side) to start review of this pull request? Or is this pull request already considered not needed (could you please decline this pull request in that case)?

Thank you.

[electron0zero]

I use OpenShift 3.11 with headless OpenShift (K8s) service. The pods, which I run in OpenShift, use TLS certificate issued by OpenShift CA. I want to use Blackbox exporter to monitor (using Prometheus rules and Alertmanager) expiration date of TLS certificates used by each pod to which the OpenShift service points. Due to OpenShift service is headless, when the name of the service (used in Blackbox exporter target) is resolved, then multiple IP addresses are returned - IP addresses of all pods to which the service points. The set and the order of resolved IP addresses remains the same b/w probes (it seems that K8s - I've tried a single node K8s with Minikube - uses round-robin DNS for headless K8s service).

@mabrarov I thought about this, and I think your use-case can be solved by having a ClusterIP service that targets same pods as headless service by using the same labels as headless service.

If we go that route, we can offload the work to ClusterIP service and don't need to make any changes in BlackBox exporter, and Blackbox exportner can just check the ClusterIP service.
This solution also allows you to have more control over the routing based on how you configure your cluster and service.

The reason I am hesitent to make this change in BlackBox exporter is because it will need maintaince forever. I am happy to change my mind if ClusterIP service doesn't work or there is good reason to have this in Blackbox exporter.

[mabrarov]

Hi @electron0zero,

Thank you for detailed response. Yes, I was thinking about non-headless service as a solution, but I was curious if Blackbox exporter itself could benefit from the new feature proposed in this pull request (and helping with my case). If you feel like these changes (which seem pretty straightforward to me - these are my first lines in Go) increase support complexity, then I'm OK with keeping Blackbox exporter (which is a nice solution!) free of them.

I appreciate if you could find time to overview the changes (if didn't do already) - if this new (but rarely needed?) feature overwhelms increased maintenance. I totally trust maintainers to make a decision in any direction.

[electron0zero]

Hi @mabrarov, thank you sending this PR.

Are there other (other then probing all pods behind a headless service) use-cases where it is useful to have in blackbox exporter?

If there are multiple use-cases, then we can consider having this in blackbox exporter, otherwise I would prefer solving k8s headless service use-case with ClusterIP service instead of baking this into blackbox exporter.

[mabrarov]

Hi @electron0zero,

I don't have other use cases, but (initially) it feels unexpected that Blackbox exporter tries just one (always the first one) IP address if probe target hostname is resolved into multiple IP addresses. It could be at least documented and at most a feature delegating the choice to the end user (like randomization) provided.

At the same time, IMHO, when domain name is resolved into multiple addresses, then it usually comes with round-robin DNS (here I'd like to rely on opinion of more experienced users, like maintainers of Blackbox exporter). So I cannot state that users of Blackbox exporter really care about chosen IP address when domain name is resolved into multiple IP addresses.

We can close (or keep opened with some resolution / mark, which clearly states the reason for that) this pull request and wait for feedback of other users of Blackbox exporter, e.g. if I need some feature in a product, then I start searching among existing issues and pull requests first - if somebody else needs the same, then he has a place to comment about that (even if pull request is declined) now.

Thank you for clarification and feedback. I really appreciate your help and efforts.

[SuperQ]

Trying the first, single, result of an address lookup is standard behavior for pretty much every application out there. Randomizing is up to the DNS server.

The fact that you are not seeing randomized results is likely a problem with your OS system configuration.

DNS round-robin has basically not worked as a valid load balancing method or reliable method of randomization. This got even worse with the introduction and implementation of RFC 3484 due to the explicit recommendation of sorting DNS lookup results by IP bits.

In addition, DNS results are cacheable. So any result randomness is subject to the whims of the caching and clients. You can not depend on it.

[SuperQ]

I'm going to leave this up to the maintainers do decide. But I am not in favor of this feature.

[electron0zero]

I am in agreement with @SuperQ. I see that this change solves a narrow problem that can be solved by using existing primitives in the k8s, and I don't see any other use-cases for this feature.

I would like to thank @mabrarov for the PR, and discussion around the topic. I would prefer not to merge this in and suggest using headless k8s service to solve the usecase, but If anything changes(like a new use-case for this feature), I am happy to reconsider ❤️