[Umbrella] Introducing `Cli` to update the configuration file

[smallzhongfeng]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [x] I have searched in the issues and found no similar issues.

Describe the proposal

Currently, our configurations are implemented using the method of scheduling threads, which has some latency. We can add cli methods, which have the following advantages:

1. Ability to refresh configuration files to memory faster
2. More secure update configuration

Task list

• [x] #770
• [ ] #769
• [x] #768
• [ ] #836
• [x] #837
• [ ] #846

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

[smallzhongfeng]

@jerqi @leixm @zuston @advancedxy If you have time, PTAL.

[smallzhongfeng]

If it's ok, I will go ahead. Gently ping. @jerqi @zuston @kaijchen @advancedxy

[zuston]

Great work. I think cli is a good way for admin to manage cluster. But do we need to trigger updating conf by CLI, the update latency is acceptable for me. Do you meet some problems on this?

[jerqi]

+1

[xianjingfeng]

As we discussed in #80, should we update conf by REST api?

[advancedxy]

If it's ok, I will go ahead. Gently ping. @jerqi @zuston @kaijchen @advancedxy

I think cli is a good way to manage shuffle clusters. I create a cli support ticket for operator to manage cluster: #569.

However your current proposal is more about to update configurations timely and securely. I'm +1 with intention. But as @xianjingfeng point out, do you think it's better to update configurations via REST api?

[smallzhongfeng]

Great work. I think cli is a good way for admin to manage cluster. But do we need to trigger updating conf by CLI, the update latency is acceptable for me. Do you meet some problems on this?

Not yet, but as I said, when updating, we need to wait with a one minute interval. However, if we use the cli method, we can take effect immediately. And in line with top-level projects, services such as Yarn and Hadoop all provide Cli for configuration updates.

[smallzhongfeng]

If it's ok, I will go ahead. Gently ping. @jerqi @zuston @kaijchen @advancedxy

I think cli is a good way to manage shuffle clusters. I create a cli support ticket for operator to manage cluster: #569.

However your current proposal is more about to update configurations timely and securely. I'm +1 with intention. But as @xianjingfeng point out, do you think it's better to update configurations via REST api?

Like Yarn, in fact, cli and REST api coexist, and I think we can follow suit.

[advancedxy]

If it's ok, I will go ahead. Gently ping. @jerqi @zuston @kaijchen @advancedxy

I think cli is a good way to manage shuffle clusters. I create a cli support ticket for operator to manage cluster: #569. However your current proposal is more about to update configurations timely and securely. I'm +1 with intention. But as @xianjingfeng point out, do you think it's better to update configurations via REST api?

Like Yarn, in fact, cli and REST api coexist, and I think we can follow suit.

co-exist works for me.

[jerqi]

If it's ok, I will go ahead. Gently ping. @jerqi @zuston @kaijchen @advancedxy

I think cli is a good way to manage shuffle clusters. I create a cli support ticket for operator to manage cluster: #569. However your current proposal is more about to update configurations timely and securely. I'm +1 with intention. But as @xianjingfeng point out, do you think it's better to update configurations via REST api?

Like Yarn, in fact, cli and REST api coexist, and I think we can follow suit.

co-exist works for me.

+1