elasticsearch
elasticsearch copied to clipboard
elastic
[DOCS] Better documentation around ILM rollover and min_age
Consider an example ILM policy like the following:
{
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "7d"
}
}
},
"warm": {
"min_age": "14d",
"actions": {
"readonly": {}
}
},
"delete": {
"min_age": "28d",
"actions": {
"delete": {}
}
}
}
}
I think it would be tempting to look at this policy and expect that an index created on 2022-04-01 with that policy would move to the warm phase on 2022-04-15 and finally be deleted on 2022-04-29. However, in an ILM policy with a rollover action, the rollover will reset the index's "age" to 0 (as reported by _ilm/explain), see https://github.com/elastic/elasticsearch/pull/30853.
Since the rollover resets the "age" to 0, a nice way of thinking about this is that the phase transition times post-rollover are relative to when the index rolled over (not to when the index was originally created).
Note, however, that in a policy without a rollover, the index "age" would not have been reset, and consequently the phase transition times would be relative to when the index was created (one caveat: unless the index.lifecycle.origination_date setting were used for that index, see the ILM settings documentation).
#84273 touched on this, specifically, it added a note onto https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html (ctrl+f for relative to).
And there's also a note on https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html itself that goes along with the last example (https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html#_rollover_condition_blocks_phase_transition), but even knowing how things work, I still find that explanation to be a bit on the subtle side.
Some places we might consider giving this top billing or at least more attention:
- https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html the high level 'conceptual' view of the index lifecycle, it mentions transitions and index ages, but not this behavior.
- https://www.elastic.co/guide/en/elasticsearch/reference/current/index-rollover.html the high level rollover documentation
- https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html the lower level rollover action documentation (n.b. to be fair, as previously stated, it is mentioned as sort of an aside in the last example on that page).
I think we could add a note to each of those pages, perhaps at the risk of being a little repetitive. However, given that this comes up fairly frequently as a point of confusion for our users, it might even make sense to go above and beyond that to provide more of a long form explainer, too, like a "here's how this works, and why it works that way" so that we give people a very good explanation on this if it happens to slip by them and then later trip them up.
The Documentation clearly misses the feature, that the rollover action resets the index age for later actions. The following statement is also a bit misleading:
Configured minimum ages must increase between subsequent phases
https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html#ilm-phase-transitions
Although it is actually correct - as the mininum age of the hot phase is always zero - the minimum age of the rollover action can be bigger than the minimum age of a later phase.
