[Segment Replication] Primary shard based rebalancing across all indices

[dreamer-89]

In #6210, primary shard balancing logic was introduced.

1. https://github.com/opensearch-project/OpenSearch/pull/6422 Added per index primary shard rebalance
2. https://github.com/opensearch-project/OpenSearch/pull/6643 introduced global primary shard balance during allocation

This change is follow up of above to implement global primary shard rebalancing logic. This can be implemented efficiently with https://github.com/opensearch-project/OpenSearch/issues/6481 implementation which switches shard roles. This change essentially needs another round of rebalancing to attain primary balance across all indices vs current which perform per index balancing.

Example use-cases

Problem 1 (Multiple primary of different index on same node)

Node 1	Node 2
P1	R3
P2	R4

Node 1	Node 2
P1	R3
	R4
	P2

Node 1	Node 2
P1	R3
R4	P2

Problem 2 (No relocation possible due to SameShardAllocator)

Node 1	Node 2
P1	R2
P2	R1

[saikaranam-amazon]

@anasalkouz Could you please assign this issue to me?

[kkewwei]

@saikaranam-amazon @dreamer-89 if not solved, I'd love to try it.

[dreamer-89]

Thanks @kkewwei for the comment. @saikaranam-amazon : Can you please confirm if you are actively working on this issue. If you are not working, do you mind assigning this issue to @kkewwei

[saikaranam-amazon]

@dreamer-89 Could you please assign to @kkewwei as I don't have permissions on the issue?

[dreamer-89]

Thanks @saikaranam-amazon for the update. @kkewwei assigned the issue to you. Please feel free to post any questions that you may have.

[Arpit-Bandejiya]

Hi @dreamer-89 @kkewwei if this is not actively worked upon, can I pick the issue?

[kkewwei]

@Arpit-Bandejiya of cause.