[FEATURE][DOCUMENTATION] Document behavior of ISM Template Priority

[bgaillard]

Is your feature request related to a problem?

We recently encounter deleted indices on one of our OpenSearch cluster and wanted to know if we have configuration errors in our ISM policies.

Looking at 2 of them we have the following configuration.

• Default policy with Index Pattern *, priority equals to 1 and transition with delete action
• Custom policy with Index Pattern foo-*, priority equals to 100 and no transition with delete action

Our problem is simple, there is no clear documentation explaining the behavior of the ism_template.priority property. The only page were I found something is the Policies one but the only mention we have is the following one.

If you don’t specify a value for priority, it defaults to 0.

What solution would you like?

An improvement of the documentation explaining clearly what's the behavior of the ism_template.priority property. Without those explanations we always worry about accidental index deletions when we have log of ISM policies with delete actions.

The improvements should explain the following.

• Is a policy with a low value having an higher priority than an other one with a high value. Or is it the inverse.
• If 2 ISM policies match the same Index Patterns what is the policy which is executed and what is it's relation to the priority
• If 2 ISM policies match the same Index Patterns are they both executed sequentially or only one of the 2 is executed

What alternatives have you considered?

Information is not easy to find without taking time to read the source code. I only found one information explaining the following here https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html

But this is not an OpenSearch documentation, it's an Elasticsearch one, and this explanation is "very light" (for example we do not know if high values are associated to high priorities of it it's the contrary).

Do you have any additional context?

Yes, if adding the documentation takes time and you have a response for the 3 points I mentioned above can you respond to them directly in a comment here ? I'm pretty sure it will help other users better understanding the behavior :)

Thanks for your help !

Baptiste

[rishabhmaurya]

Is a policy with a low value having an higher priority than an other one with a high value. Or is it the inverse.

High value indicated higher priority and not the inverse as stated here - policies

I agree that this setting is quite confusing with lack of documentation. Adding a documentation label to the issue.

[bgaillard]

Hi @rishabhmaurya, thanks for the information about the priority.

The link you provided is not exactly linked to the policy.ism_template.priority property so it was not obvious that the priority behavior is the same as the one used in "operations".

In addition can you ensure me that if 2 ISM policies match the same Index Patterns then only the policy with the highest priority will be executed and the other will not be executed ?

For exemple in our case we have the following policies, can we be 100% sure the "default" one is not executed on an index named "sample-1" ?

# Policy "default"
{
   ...
   "ism_template": [
     {
       "index_patterns": ["*"],
       "priority": 1
     }
   ]
}

# Policy "sample"
{
  ...
  "ism_template": [
    {
      "index_patterns": ["sample-*"],
      "priority": 201
    }
  ]
}

Thanks :)

[bowenlan-amzn]

Hi @bgaillard, it's for sure among all the policies that matches the newly created index, only the one with highest priority will be used to manage this index. https://github.com/opensearch-project/index-management/blob/f543d930ab698157967e38d9e474717d2ea45bd4/src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt#L357

Also, I will update our document about the ism_template.priority in policy.

[bgaillard]

Hello @bowenlan-amzn, thanks for your response and the link you provided.

There is something which seems strange to me in the source code, the and is created before in dex creation date one.

It seems confusing to me and probably a little "magic" for the users. If a user creates a policy after an index is created to work on this index then it will simply be not elected ?

In our case we're sadly on an old Opensearch version and I'm not sure the source code associated to our version is mapped to the link you shared.

We finally changed all our policies in such a way that it's never possible to have 2 policies which can be elected for the same index. If this is the case we have a control code which throws an exception in our CI so it's never possible to be in that case. If we have an index with no policy available for this index we also throw an exception. With this setup we cover all the indices in our clusters and are 100 % sure we correctly manage retention for them (our clusters are big ones used to store application logs).

Anyway all those misunderstanding confirm that the documentation would need clarification on the exact behavior about policy executions.

Thanks 🙂

[bowenlan-amzn]

Yes, the ism_template behavior is consistent with index template function provided in OpenSearch core, it won't change already created index, but only work when new index created.

The control code parts sounds cool 👍