blackbox_exporter icon indicating copy to clipboard operation
blackbox_exporter copied to clipboard

Add retry logic to DNS prober

Open mem opened this issue 1 year ago • 2 comments

With DNS, since it's possible use UDP traffic, depending on the relative network location of the server under test, sometimes packets are dropped and the probe ends up timing out. It becomes hard to distinguish between this and an actual failure in the server.

Introduce an optional retry mechanism where requests are retried for a limited number of times. Each request gets its own timeout that still respects the timeout configured for the probe.

For example, if you configure 3 retries, each of them with a 3 second timeout, and the timeout for the probe is 5 seconds, the first attempt might consume 3 seconds, leaving 2 seconds until the deadline. The second request will still use a timeout of 3 seconds, but the probe itself will timeout and it will fail as usual.

Add a new metric that reports the number of attempts. Note that the probe_duration_seconds metric will now report the total time until an answer was obtained. In the above example, if the second attempt produces a response within 1 second, the reported duration will be ~ 4 seconds. The rtt will be ~ 1 second. Both these metrics allow the user to differentiate and track requests that require continous retries, indicating some kind of issue with the server or the network path.

mem avatar Jul 19 '24 23:07 mem

@SuperQ I would appreciate your opinion here, particularly regarding the metrics. I tried explaining the case that I'm trying to deal with.

mem avatar Jul 19 '24 23:07 mem