direct link churning

[real-danm]

Nebula prefers direct links to hosts, however, in situations with asymetric routing or links with high packet loss as described in https://github.com/slackhq/nebula/issues/893 the direct link will churn forever leading to decreased throughput or even no connectivity at all. This PR attemtps to throttle direct link churning and successfully mitigates situations with asymetric routing or links with high packet loss in conjuction with a few other settings.

In a 3-node set up (client-1, client-2, relay/lighthouse) where client-1 has either a full ip-block to client-2 or an 80% loss with iptables -A INPUT -s {client-2-pub-ip} -m statistic --mode random --probability 0.8 -j DROP

Setting the following values will lead to very quick switch to relays; with a retry of the direct link in 60s

handshakes:
  churn_limiting: true
  churn_num_failures: 1
  churn_period: 60s
timers:
  connection_alive_interval: 1
  pending_deletion_interval: 1

[salesforce-cla[bot]]

Thanks for the contribution! Before we can merge this, we need @real-danm to sign the Salesforce Inc. Contributor License Agreement.

[ExplodingDragon]

Any new developments on this？ Right now it conflicts with main

[real-danm]

Any new developments on this？ Right now it conflicts with main

I have moved on to a different (private) fork with additional features. It doesn't seem like this will get merged; haven't received any feedback. If you want to use this fork, i'm happy to clean up the conflicts, just let me know.

[johnmaguire]

Hi Dan, thanks for the filed issue and this PR. This isn't a scenario we've run into yet, and you seem to be the first user to report it. While I'm sure it's a legitimate scenario, we haven't had enough experience or time to think through the causes or what an ideal solution looks like. Specifically we are a bit gun shy to add additional state that looks similar to how ConnectionManager behaves to detect and stop churn.

Seeing as you've moved on to your own fork, we're going to close this for the time being.

@ExplodingDragon If you're experiencing issues, please file a new issue describing what you're seeing.