[AMORO-2810]: Support multiple nodes to access Amoro Rest service in a high availability environment

[czy006]

Support multiple nodes to access Amoro Rest service in a high availability environment

Why are the changes needed?

• Support multiple nodes to access Amoro Rest service in a high availability
• Just Leader node running OptimizingService

Close #2810.

Brief change log

• Support multiple nodes to access Amoro Rest service in a high availability environment

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [x] Add screenshots for manual tests if appropriate

• [x] Run test locally before making a pull request

Documentation

• Does this pull request introduce a new feature? (yes)
• If yes, how is the feature documented? (not documented)

[czy006]

3 amoro nodes,normal change service to leader

[codecov-commenter]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 21.76%. Comparing base (e69710a) to head (55b63e4). :warning: Report is 58 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #3567   +/-   ##
=========================================
  Coverage     21.76%   21.76%           
  Complexity     2391     2391           
=========================================
  Files           436      436           
  Lines         40498    40498           
  Branches       5743     5743           
=========================================
  Hits           8816     8816           
  Misses        30935    30935           
  Partials        747      747           

Flag	Coverage Δ
trino	21.76% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[klion26]

Sorry for jumping in this late.

After this change, will there be two Amoro servers handle the same table update(put/post etc rest) logic? If this is yes, will this lead to some wrong result/state?

[czy006]

Sorry for jumping in this late.

After this change, will there be two Amoro servers handle the same table update(put/post etc rest) logic? If this is yes, will this lead to some wrong result/state?

Optimization services will not be processed simultaneously, we ensure that only one master node handles Optimizer. Other child nodes can handle Rest requests, such as creating and deleting catalogs. Will this have an impact?

[klion26]

Sorry for jumping in this late. After this change, will there be two Amoro servers handle the same table update(put/post etc rest) logic? If this is yes, will this lead to some wrong result/state?

Optimization services will not be processed simultaneously, we ensure that only one master node handles Optimizer. Other child nodes can handle Rest requests, such as creating and deleting catalogs. Will this have an impact?

As there is more than one server that will update the same table's info/state, we need to handle this carefully to avoid the inconsistent state of concurrent modification. Maybe we need to add a doc to analysis this

[zhoujinsong]

Rplaced by #3737.