Add primary preference to be set as an index setting for the warm index

[neetikasinghal]

For a writable warm index, the default search preference should be set to primary such that the traffic is routed to the primary shards always for a warm index.

Currently, the search preference can be set only at a request level, so if a user doesn't specify the search preference as primary, the request can be routed to both primary/replica shards.

Proposal here is to create an index level setting to set the search preference as "_primary_first" by default for a warm index. ref: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/cluster/routing/Preference.java#L69

[peternied]

[Triage - attendees 1 2 3] @neetikasinghal Looks like a work item in the storage ecosystem

[neetikasinghal]

Checking more on how searchable snapshot deals with it -

Its dealt with in the search path itself rather than introducing a new index setting for the warm index: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/cluster/routing/OperationRouting.java#L244

Something similar can be done even for the writable warm index as well. This mechanism also honors the preference specified by the end user instead of always defaulting the primary preference. However it would be worth assigning the default value to _primary_first instead of primary

[linuxpi]

[Storage Triage - attendees 1 2 3 4 5 6 7 8]

@neetikasinghal Thanks for filing this issue, please feel free to submit a pull request and add more details