[Feature Request] Background Tasks

[linuxpi]

Is your feature request related to a problem? Please describe

With Offline fleet, data nodes and Offline Fleet nodes itself can submit background tasks to Offline Fleet. At any point, the no of Tasks submitted might be too much for available nodes in the Offline Fleet to distribute amongst themselves.

Even if we do try to assign a task to a particular node right after its submitted, the node may or may not have resources at that time to start the task and would need to put it into a “Queue”. Apart from that, if the node goes down, and this Queue is not persisted in Remote, all those tasks in Queue are lost.

Describe the solution you'd like

To support various background tasks to be run on dedicated/separate fleet of nodes, we need to explore and propose ideas around how to model tasks and submit for execution of the dedicated fleet

RFC - https://github.com/opensearch-project/OpenSearch/issues/12361

Related component

Storage

Describe alternatives you've considered

No response

Additional context

No response

[linuxpi]

Design Details outlined in https://github.com/opensearch-project/OpenSearch/issues/13554

[getsaurabh02]

@linuxpi do we need to move this to 2.17?