[Feature Request] Support deploy multiple shuffle servers in a single node

[xianjingfeng]

We have 1.2T memory per host, so we need deploy multiple shuffle servers in a single node. But current partition assignment policy is not suitable to do that. Because partition maybe assign to a same host and client maybe fail if one host down

[zuston]

As i know, partition assignment policy could be implemented by yourself.

[xianjingfeng]

As i know, partition assignment policy could be implemented by yourself.

Yes, but not only assignment policy need to be change, such as start script, i will create a pr for this

[jerqi]

I'm also curious why we need to modify start script?

[xianjingfeng]

I'm also curious why we need to modify start script?

start script will process existence

[xianjingfeng]

If our plan is deploy on k8s, this issue should close?

[jerqi]

K8S Deployment can't solve this issue.

[colinmjj]

I'm also curious why we need to modify start script?

start script will process existence

Yes, this is the limit with current implementation

[jerqi]

K8S Deployment can't solve this issue.

@xianjingfeng I think you can continue this.

[xianjingfeng]

K8S Deployment can't solve this issue.

@xianjingfeng I think you can continue this.

ok