[Searchable Snapshot] Design a new ShardsAllocator to handle remote shards

[kotwanikunal]

Is your feature request related to a problem? Please describe.

• Searchable snapshots will introduce a new remote searcher role - https://github.com/opensearch-project/OpenSearch/issues/3895
• As a part of this, we would like to design a new shard allocation logic which will assign the searchable snapshot shards to the nodes with this new role
• There is a POC implementation which was done as a part of #4420 and exists at https://github.com/kotwanikunal/OpenSearch/pull/4

Describe the solution you'd like

• Design for a new ShardsAllocator which ensures that the remote shards are assigned to the nodes with a remote_searcher role (Ref: https://github.com/kotwanikunal/OpenSearch/pull/4)
• Ensures that the hybrid nature of having both the local and the remote shards on the single node are handled appropriately wherein the rebalancing logic for the local shards does not affect the remote shards, remote shards have the appropriate recovery process to handle node failures (Handled by the POC using basic checks, will need a deeper dive)
• The AllocatorDecider(s) are capable of handling both the remote and local shards for the hybrid setup (Ref: https://github.com/kotwanikunal/OpenSearch/pull/4)

Additional context

• #4420
• #4306