incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

Published 20 hours ago • verified publisher icon apache

[Feature Request] Support shuffle server decommissioned

Open xianjingfeng opened this issue 3 years ago • 10 comments

kill process is not graceful, so we need shuffle server support decommissioned

xianjingfeng avatar Jul 26 '22 10:07 xianjingfeng

+1. Now the decommission could be used by exclude node file in coordinator side.

Besides, the exclude-node-file could be stored in HDFS.

zuston avatar Jul 26 '22 10:07 zuston

I understand that you need a rolling upgrade feature. In our plan, we want to accomplish this feature by k8s operator. For the standalone mode, we don't have the plans,. And it's also necessary to do some surveys about this feature, I think we should have more discuss about this problem.

jerqi avatar Jul 27 '22 04:07 jerqi

I understand that you need a rolling upgrade feature. In our plan, we want to accomplish this feature by k8s operator. For the standalone mode, we don't have the plans,. And it's also necessary to do some surveys about this feature, I think we should have more discuss about this problem.

Depoly on k8s is a good choice. But one more choice is not a bad thing. Not all teams willing to use k8s . I have create a pr.

xianjingfeng avatar Jul 27 '22 12:07 xianjingfeng

Could you write a design doc's (use google doc) ? Because this issue is a little complex.

jerqi avatar Jul 27 '22 14:07 jerqi

If we want to add some interface to control shuffle server's behavior, we should have a complete design, and we think we need detailed discussions. We ever have similar mind in issue #37

jerqi avatar Jul 27 '22 15:07 jerqi

Yarn node's decommission. https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html Maybe, we should also realize other system decommission implement.

jerqi avatar Jul 28 '22 16:07 jerqi

Yarn node's decommission. https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html

Yes. In #85, I follow the rule of YARN decommission mechanism. So i think it's better to control the decommission by coordinator. Feel free to discuss more

Maybe, we should also realize other system decommission implement.

I looked the HDFS datanode decommission, it's also like YARN decommission. refer: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html

zuston avatar Jul 29 '22 02:07 zuston

I think we should consider more things, such as

  1. Is it easy to use if we deploy on k8s and IP is not fixed?
  2. Split-brain. If pass commands through heartbeat, shuffle server may receive different messages meanwhile and how do we ensure correctness. The decommission function can be solved. What about other functions in the future?
  3. Compatibility. If pass commands through heartbeat, we need to modify this interface frequently.

xianjingfeng avatar Jul 31 '22 01:07 xianjingfeng

cc @colinmjj . What do you think? I remember that you want to use coordinator to dispatch the configuration to shuffle servers. It's similar to use coordinator to decommission.

jerqi avatar Jul 31 '22 14:07 jerqi

I think such feature is about command line or some API to manage the behavior of coordinator/shuffle server. There should be an overall picture to describe how to make this happen. Besides decommission, how about update some configuration in shuffle server, clear shuffle data(which maybe useful for streaming jog), etc. All above feature is management related, so I prefer to have a framework which can involve all these things.

colinmjj avatar Aug 01 '22 02:08 colinmjj

@jerqi @colinmjj I want to know if you have plan recently. We have some functions that need to be built on the function of decommission, such as auto scaling We don't want to deviate too much from the community.

xianjingfeng avatar Oct 24 '22 12:10 xianjingfeng

@jerqi @colinmjj I want to know if you have plan recently. We have some functions that need to be built on the function of decommission, such as auto scaling We don't want to deviate too much from the community.

We have no related plan recently. If you have interest about this topic, we can start a offline meeting to discuss this issue.

jerqi avatar Oct 25 '22 02:10 jerqi

@jerqi @colinmjj I want to know if you have plan recently. We have some functions that need to be built on the function of decommission, such as auto scaling We don't want to deviate too much from the community.

We have no related plan recently. If you have interest about this topic, we can start a offline meeting to discuss this issue.

+1.

zuston avatar Oct 25 '22 04:10 zuston

we can start a offline meeting to discuss this issue.

I am looking forward to it.

xianjingfeng avatar Oct 25 '22 05:10 xianjingfeng

@zuston @xianjingfeng There are some other issues which we need to discuss, so I will send a email to our dev mail list, and select a proper date to start the meeting.

jerqi avatar Oct 25 '22 06:10 jerqi

@xianjingfeng I have already send an email https://lists.apache.org/thread/2jlm3fswmsxy619ldyo4px700p3ybnvc. Do you have time at 11 am (UTC +8) Thursday this week?

jerqi avatar Oct 26 '22 03:10 jerqi

@xianjingfeng I have already send an email https://lists.apache.org/thread/2jlm3fswmsxy619ldyo4px700p3ybnvc. Do you have time at 11 am (UTC +8) Thursday this week?

Yes, i have time.

xianjingfeng avatar Oct 26 '22 04:10 xianjingfeng

@xianjingfeng I have already send an email https://lists.apache.org/thread/2jlm3fswmsxy619ldyo4px700p3ybnvc. Do you have time at 11 am (UTC +8) Thursday this week?

Yes, i have time.

Meeting link is https://meeting.tencent.com/dm/oR95wASCNe91

jerqi avatar Oct 26 '22 06:10 jerqi

@xianjingfeng I have already send an email https://lists.apache.org/thread/2jlm3fswmsxy619ldyo4px700p3ybnvc. Do you have time at 11 am (UTC +8) Thursday this week?

Yes, i have time.

Meeting link is https://meeting.tencent.com/dm/oR95wASCNe91

Get

xianjingfeng avatar Oct 26 '22 06:10 xianjingfeng

Offline Discussion Result: Coordinator provide admin rest api, Coordinator is only used as proxy, Coordinator redirect the request to shuffle server by rpc. Currently, we need the api

  1. Decommission
  2. UpdateConfiguration
  3. Upgrade

cc @zuston, you can give your advice for us.

jerqi avatar Oct 27 '22 04:10 jerqi

Design doc: https://docs.google.com/document/d/1p1PksBN2LJ-OtGEHvdyEuH9b1Mv1aD_exMPl4TNaTs0/edit?usp=sharing PTAL @jerqi @zuston

xianjingfeng avatar Jan 09 '23 10:01 xianjingfeng

Thanks a lot for proposing this, I will take a look ASSP

zuston avatar Jan 09 '23 10:01 zuston

Design doc: https://docs.google.com/document/d/1p1PksBN2LJ-OtGEHvdyEuH9b1Mv1aD_exMPl4TNaTs0/edit?usp=sharing PTAL @jerqi @zuston

Commented. @xianjingfeng PTAL

zuston avatar Jan 09 '23 10:01 zuston

As we discuss in design doc, i will make the following adjustments:

  1. Add concrete rest api list in the design doc.
  2. Remove token in this design.

Any other suggestions? @jerqi @zuston @advancedxy

xianjingfeng avatar Feb 01 '23 02:02 xianjingfeng

I'm OK.

jerqi avatar Feb 01 '23 02:02 jerqi

As we discuss in design doc, i will make the following adjustments:

  1. Add concrete rest api list in the design doc.
  2. Remove token in this design.

Any other suggestions? @jerqi @zuston @advancedxy

+1. Thanks for your effort

zuston avatar Feb 01 '23 04:02 zuston

Thanks @xianjingfeng for working on this feature. I'm closing this issue now. Please feel free to reopen it if there is more work.

kaijchen avatar Mar 10 '23 08:03 kaijchen