Automatically update raft IP address when it changes

[aaronlehmann]

This adds an AddrMismatch flag to ProcessRaftMessageResponse, which allows any node that receives a Raft message to signal to the sender that its address didn't match the expected value. Based on this, the sending node can try to call Join again to update its address in Raft.

Using a flag returned on responses to Raft messages allows us to notice address changes that happen at any time, not just between daemon restarts.

This behavior will only be enabled when the advertise address was originally autodetected, to prevent fixed addresses from being changed. A flag has been added to inform the raft code whether the address was autodetected or not. The only change necessary in Docker Engine to enable this feature will be to set this flag in cases where the advertise address was originally autodetected.

DEPENDS ON #2198

cc @aluzzardi @cyli

[codecov[bot]]

Codecov Report

Merging #2199 into master will increase coverage by 0.13%. The diff coverage is 45%.

@@            Coverage Diff             @@
##           master    #2199      +/-   ##
==========================================
+ Coverage    60.1%   60.23%   +0.13%     
==========================================
  Files         128      128              
  Lines       26177    26204      +27     
==========================================
+ Hits        15734    15785      +51     
+ Misses       9045     9015      -30     
- Partials     1398     1404       +6

[aaronlehmann]

I have corresponding moby changes at https://github.com/aaronlehmann/docker/tree/swarmkit-rejoin

I've tested this with a cluster and verified that when I restart machines and their IPs change, they get updated automatically.

This branch also has the changes to support swarm join while already part of a cluster. I haven't tested this yet, though, and I'm starting to question the usefulness of this part.

[aluzzardi]

LGTM

[thaJeztah]

needs a rebase

[cyli]

I can rebase and carry this forward, if that would be helpful :)

[aaronlehmann]

I would appreciate that! But keep in mind the following:

• This feature would be incomplete if overlay networking and gossip can't handle IP changes. Before merging it, you may want to make sure that those handle the situation correctly, or wait until the necessary changes are made.
• You'll need to carry the moby changes for this change to be useful.
• There was some UX discussion that never seemed completely resolved. They may pop back up if/when a moby PR is opened.

[abhi]

@cyli I have the corresponding network changes. I haven't gotten a chance to test it out end to end. I will try it this week

[cyli]

@aaronlehmann Thanks for the caveats! I didn't realize that networking and gossip may not handle the changes. Will do. And thanks @abhinandanpb - was just about to ping you :)

[cyli]

(I can't seem to push to this branch - I will open another PR that references this whenever @abhinandanpb would find it useful)

[aaronlehmann]

I've opened this branch to other committers.

[cyli]

Thanks @aaronlehmann!

[cyli]

This LGTM if everything it passes @abhinandanpb's network tests.

[thaJeztah]

ping! what the status on this one?

[cyli]

@abhi Have the network changes been merged? If so, if I rebase this to the latest master and the tests pass, is this good to go on rebase?

[thaJeztah]

ping @abhi ^^

[orenye]

Hi, Any news on this issue?

[nstankov-bg]

This is a much needed feature. Can it please undergo review and rebase ? @thaJeztah @aaronlehmann

[aaronlehmann]

Wow, interesting to see this PR is still open after several years. I'm no longer involved in the swarmkit project, so I think one of the current maintainers would have to pick this up and bring it up to date.