[Feature Request] Support shuffle server decommissioned

[xianjingfeng]

What changes were proposed in this pull request?

1. Create a new rpc api for decommissioned in shuffle server, we can request this api by shell.
2. When shuffle server is in decommissioned state, it will exit where all applications finished and it will report to coodirator by heartbeat.
3. Coodirator will exclude decommissioned node when assign partition.

Why are the changes needed?

kill process is not graceful, so we need shuffle server support decommissioned

Does this PR introduce any user-facing change?

No

How was this patch tested?

tests were added

[zuston]

I'm confused why not reuse the exclude-node-file? And the decommission operation maybe controlled by coordinator will be better?

Do u have any ideas on it? @jerqi

[jerqi]

I'm confused why not reuse the exclude-node-file? And the decommission operation maybe controlled by coordinator will be better?

Do u have any ideas on it? @jerqi

Agree with you.

[xianjingfeng]

1.Exclude is not equals to decommission, we can refer to HDFS 2.Coordinator should be stateless, if not, we need ensure multi instance data synchronization 3.I don't think store exclude-node-file in hdfs and update periodically is a good choice. 4.Whether or not controlled by coordinator，shuffle server need to expose an api to respond to this event. we can support controll by coordinator base on this pr 5.Maybe we can support this function first. I think without this function, users not very willing run uniffle in the production environment. Rome wasn't built in a day, uniffle has many shortcomings now 6.I hope you have better idea and implement it

[jerqi]

1.Exclude is not equals to decommission, we can refer to HDFS 2.Coordinator should be stateless, if not, we need ensure multi instance data synchronization 3.I don't think store exclude-node-file in hdfs and update periodically is a good choice. 4.Whether or not controlled by coordinator，shuffle server need to expose an api to respond to this event. we can support controll by coordinator base on this pr 5.Maybe we can support this function first. I think without this function, users not very willing run uniffle in the production environment. Rome wasn't built in a day, uniffle has many shortcomings now 6.I hope you have better idea and implement it

1. Although node exclusion is not equals to node decommission, the two operation have some similar mechanism, I think we should maintain one mechanism.
2. Agree with you, should we add a extra service for standalone mode deployment?
3. Agree with you, better solution is to have configuration abstraction.
4. Maybe some people think we can reuse the heartbeat to control the shuffle server, it depends our concrete implement.
5. We should have design document , discuss about more details, do more survery about other system. We need this function, but this implement may not be the best solution.