[FEATURE] Copy metadata from last index during rollover

[soosinha]

Is your feature request related to a problem? Follow up from the feature request in OpenSearch: https://github.com/opensearch-project/OpenSearch/issues/8419

During rollover, the index metadata is picked up from the template. But in case the mappings have been updated during ingestion due to dynamic mappings being enabled, the new index will not have the latest mappings. When data is ingested on the new index, there will be updation of mappings immediately which will result in cluster state updates. These cluster state updates can potentially increase the load on the cluster manager nodes.

What solution would you like? Rollover action should have an option to copy the metadata from the previous index. This way the mappings will be in sync with documents being ingested and we can avoid redundant dynamic mappings updates.

Detailed API level changes: The rollover action will have a new boolean parameter use_previous_index_metadata like below:

{
  "rollover": {
    "use_previous_index_metadata": true
  }
}

This parameter will be false by default. When this is set to true, during rollover the metadata will be copied from the last write index of the alias or the data stream.

[soosinha]

@opensearch-project/index-management Let me know if I can go ahead with this change.

[bowenlan-amzn]

@soosinha please go ahead. We have a request to copy over all aliases from previous index to newly created index during rollover, will your enhancement also cover aliases or it will only cover mappings of index metadata.

[soosinha]

@bowenlan-amzn Thanks for the confirmation. Can you assign this issue to me?

The plan was to copy only settings and mappings from the previous index. But the issue that you have linked here seems very similar to current scope. I should be able to handle the copying of aliases as well in this change.

Should one parameter like copy_metadata be sufficient to control the copying of aliases, settings and mappings or do we need separate parameters like copy_alias and copy_mappings ?

[bowenlan-amzn]

Thanks a lot! I would suggest using separate parameters for aliases, settings and mappings.

[JahanviAcquia]

any update on this ticket? We have dynamic attribute mapping. So we will need to update rollover alias manually