Akka/Pekko retry frenzy when all hosts are blacklisted

[mirzov]

It appears that the retry logic in AkkaHttpClient has a problem in the scenario when all the hosts got blacklisted. The problem is that in this situation retry is attempted without any delay, and the retry is repeated until maxRetryTimeout time has passed since the first attempt, but the repetition is doomed to failure because all the hosts are blacklisted. This results in high CPU usage, but without any network calls to Elastic. Perhaps the problem can be solved by adding the extra condition for going into another retry, namely checking that there are non-blacklisted hosts? Something like

if (timePassed < settings.maxRetryTimeout.toNanos && settings.hosts.exists(!blacklist.contains(_)) {
   logger.trace(s"Retrying a request: ${request.endpoint}")
   queueRequestWithRetry(request, startTimeNanos)
}

[Philippus]

In https://github.com/Philippus/elastic4s/pull/1767 it was specifically added to retry even if all hosts are blacklisted. It seems to me that retrying when all hosts are blacklisted only makes sense if some delay is built in.