[streams][lifecycle] update `inherit` behavior for classic streams

[klacabane]

trafficstars

Follow up from https://github.com/elastic/kibana/pull/221364#issuecomment-2935982090

Currently calling Reset to default action (inherit option) on an unwired stream will do nothing, it just means that streams will stop managing the lifecycle and it can be updated outside of the APIs. As a result the lastly applied lifecycle will remain in place.

This is both not that useful and unexpected, instead one would expect the initial configuration (before onboarding the stream) to be reverted

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[klacabane]

Options for reset to default

1. revert back to the configuration set at the time on onboarding - when we first persist the unwired stream, we save the current data stream lifecycle configuration and apply it back when the option is selected. One risk is that the configuration points to an ilm policy that may not exist anymore. While the UI shows that error, a change via the API would go through without error. An additional side effect is that the revert may not be completely true to its initial configuration: when we apply a dsl or ilm via stream we update all the backing indices so we get a consistent retention throughout the data. An unwired stream at the time of onboarding could have mixed retention throughout its backing indices that we won't be able to restore
2. revert back to the template's configuration - apply configuration set on index/component template. One risk is that we'd lose any datastream-level configuration set by users before onboarding (+ the risks about deleted ilm and mixed configuration mentioned in 1.). This could be explicitly mentioned, and the option named Reset to template configuration so there's no magic. One note on this option is that with the current APIs the logic would be 1. figure out what is the index template lifecycle configuration 2. apply the configuration on the data stream. The implication is that, if the index template changes its lifecycle configuration, the new settings won't be picked up by the data stream because we've set an override. If we need the datastream to follow the template's configuration (ie new backing indices are created with the template's configuration) we'd need a way to delete the overrides which I think is not doable at the moment (I didn't look too much into it yet, but maybe @masseyke knows a way?)
3. do not offer this option at all for unwired (current behavior) - although the stream starts in an inherit state, it is not possible to revert back to inherit once updated to dsl or ilm via streams

I'm leaning towards 2. if we can actually fall back to the live template's configuration, otherwise 3. sounds reasonable. @flash1293 @LucaWintergerst Thoughts ?

[flash1293]

I would like number 2 as well, but without using an override, that sounds confusing. There should probably be a way to remove the override and revert to what the template dictates anyway, even for non-stream use cases. But interested in what @LucaWintergerst and @masseyke think

[LucaWintergerst]

I would also go with number 2 - that seems to be the most predictable even though it has some caveats.

[flash1293]

@klacabane since you self-assigned, are you going to pick this up?

[klacabane]

@flash1293 yes, I've looked into how we could reset data stream level settings to fallback to the one configured at the template level but couldn't find a way with the current GET, PUT apis (eg index.lifecycle.name: null would still take precedence) so I think we need a DELETE or a way to remove the data stream settings.

@masseyke is there currently a way to fallback to the template settings once data stream settings have been set ?

[flash1293]

@klacabane isn't this just

PUT _data_stream/logs-generic-default/_settings
{
  "index": {
    "lifecycle": {
      "name": null
    }
  }
}

or am I missing something here?

[klacabane]

(eg index.lifecycle.name: null would still take precedence)

@flash1293 I think index.lifecycle.name: null is considered a valid option, and in that case no lifecycle would be applied:

// created with default logs lifecycle
POST logs-lifecycle-overwrite/_doc
{
    "@timestamp": "2025-06-16"
}

// new write index picks up `logs` lifecycle
POST logs-lifecycle-overwrite/_rollover

// overwrite lifecycle at data stream level
PUT _data_stream/logs-lifecycle-overwrite/_settings
{
    "index.lifecycle.name": ".alerts-ilm-policy"
}

// new write index picks up `.alerts-ilm-policy` lifecycle
POST logs-lifecycle-overwrite/_rollover

// attempt the remove overwritten setting
PUT _data_stream/logs-lifecycle-overwrite/_settings
{
    "index.lifecycle.name": null
}

// new write index is `Unmanaged`
POST logs-lifecycle-overwrite/_rollover

[flash1293]

Oh, I see, good point... Looks like we need a follow-up here, waiting for @dakrone and @masseyke

[dakrone]

I think it would be beneficial to have a way to remove an override, @masseyke what do you think?

[flash1293]

FWIW, what I tried out of instinct is

PUT _data_stream/logs-generic-default/_settings
{
  "index": {
    "lifecycle": null
  }
}

but it didn't work

[masseyke]

I think it would be beneficial to have a way to remove an override, @masseyke what do you think?

Yeah it definitely would be. Should setting it to null just remove the setting? Or is there a case where a user really would want to say "yeah I know the template has a value for this setting but I really just want it to be unset"? Maybe we need a delete data stream setting API?

[flash1293]

Since we anyway have a reconciliation mechanism on the Kibana side we can represent a different API call for deleting/unsetting compared to setting without too much hassle, but IIRC it would be easier for us to make it part of the PUT API call - what do you think @klacabane ?

[klacabane]

I think we'll have to handle this as a special case in Kibana so it should not matter how it is implemented, but I wonder if we have any example of such behavior in ES api where setting a null value means ignore the value ? I'll have to check but how template inheritance works, if a higher precedence component template has a setting value of null, does that setting gets picked up in the lower precedence template or does it remains null ? If the former we'd be somehow consistent by passing nulls to the current PUT _settings api, otherwise a DELETE api makes more sense ?

[flash1293]

PUT myindex/_settings
{
    "refresh_interval": "5s"
}

GET myindex/_settings

// returns
{
  "myindex": {
    "settings": {
      "index": {
        "refresh_interval": "5s"
      }
    }
  }
}

PUT myindex/_settings
{
    "refresh_interval": null
}

GET myindex/_settings

// returns
{}

is kinda like that.

PUT myindex/_settings
{
    "refresh_interval": null
}

would also preserve some other setting that is present on the index - refresh_interval: null just removes that one setting. It feels natural to me

[masseyke]

To summarize an offline conversation, we'll go with the suggestion above -- putting a null value for a setting will remove the setting from the data stream, and result in the setting value being pulled from the template (or system default if there is not value in the template). There will be no way to set the value of a setting to null on a data stream since setting it to null in the API will cause it to be removed from the data stream. There will be no new delete API.

[masseyke]

I've put up a PR for this at https://github.com/elastic/elasticsearch/pull/129677

[flash1293]

The Elasticsearch PR for this got merged, so we can build this feature now. Won't be in 9.1, but that's OK. I'm adding it to the milestone

[klacabane]

@masseyke passing null to prefer_ilm fails, is there a limitation on the property types that we can unset ?

PUT _data_stream/logs-foo-bar/_settings
{
  "index.lifecycle.prefer_ilm": null
}
---->

{
  "data_streams": [
    {
      "name": "logs-foo-bar",
      "applied_to_data_stream": false,
      "error": """Cannot invoke "org.elasticsearch.cluster.metadata.Template.settings()" because the return value of "org.elasticsearch.cluster.metadata.ComposableIndexTemplate.template()" is null""",
      "settings": {},
      "effective_settings": {},
      "index_settings_results": {
        "applied_to_data_stream_only": [],
        "applied_to_data_stream_and_backing_indices": []
      }
    }
  ]
}

~@flash1293 When we unset values from data stream settings, the fallback values from the template won't be automatically applied to the backing indices and only the next write index will pick up the new lifecycle. This is inconsistent with the feature behavior but we can workaround this with a couple extra steps (alternatively this could be handled by the _settings API ?):~ ~- unsetting values~ ~- determine the new data stream lifecycle~ ~- apply it only on the backing indices~

my bad, it actually applies the fall back values to the backing indices

[masseyke]

@masseyke passing null to prefer_ilm fails, is there a limitation on the property types that we can unset ?

Hmm, the test I wrote tests 2 of the 3 settings (with prefer_ilm being the one left out): https://github.com/elastic/elasticsearch/blob/main/modules/data-streams/src/yamlRestTest/resources/rest-api-spec/test/data_stream/240_data_stream_settings.yml#L360-L367. Maybe for some strange reason prefer_ilm does not behave the same? I'm about to add a test for it.

[masseyke]

@klacabane it looks like setting prefer_ilm to null works fine when I add that to my test, so I'm missing some step. Can you give me some rest calls to reproduce this? Based on the error message it looks like somehow you have a null template, and I don't know how to make that happen.

[klacabane]

@masseyke

// 1. create data stream that picks up the builtin `logs` template
POST logs-foo-bar/_doc
{
    "@timestamp": "2025-06-16"
}

// 2. set overrides
PUT _data_stream/logs-foo-bar/_settings
{
  "index.lifecycle.name": "foo",
  "index.lifecycle.prefer_ilm": true
}

// 3. unset overrides
PUT _data_stream/logs-foo-bar/_settings
{
  "index.lifecycle.name": null,
  "index.lifecycle.prefer_ilm": null
}

--->

{
  "data_streams": [
    {
      "name": "logs-foo-bar",
      "applied_to_data_stream": false,
      "error": """Cannot invoke "org.elasticsearch.cluster.metadata.Template.settings()" because the return value of "org.elasticsearch.cluster.metadata.ComposableIndexTemplate.template()" is null""",
      "settings": {},
      "effective_settings": {},
      "index_settings_results": {
        "applied_to_data_stream_only": [],
        "applied_to_data_stream_and_backing_indices": []
      }
    }
  ]
}

Note that passing index.lifecycle.name: null before setting overrides will produce a similar error, but prefer_ilm appears to fail all the time

[masseyke]

OK thanks. I can reproduce it now. Working on a fix.

[masseyke]

Here's the fix: https://github.com/elastic/elasticsearch/pull/130394

[masseyke]

I think there is a similar problem for mappings. I'm currently working on that.

[klacabane]

Falling back to the template's lifecycle configuration works when the template defines an ILM but I can't find a way to fall back to a template's DSL value once it has been updated with another data retention value:

// template with an enabled DSL
PUT _index_template/test_dsl_fallback
{
    "index_patterns": ["foobar-*"],
    "data_stream": {},
    "template": {
        "lifecycle": { "data_retention": "5d" },
        "settings": {
            "index.lifecycle.prefer_ilm": false
        }
    }
}

// data stream is configured to keep data for 5d
POST foobar-baz/_doc
{
    "@timestamp": "2025-07-02"
}

// data stream is configured to keep data for 100d
PUT _data_stream/foobar-baz/_lifecycle
{
  "data_retention": "100d"
}

// how to go back to the template's value ?
DELETE _data_stream/foobar-baz/_lifecycle // <- this sets the data stream as Unmanaged

PUT _data_stream/foobar-baz/_lifecycle
{} // <- this sets the data stream with an infinite retention

PUT _data_stream/foobar-baz/_lifecycle
{
  "data_retention": null  // <- this is invalid
}

@masseyke is there a way to unset a DSL similar to the settings ?

[masseyke]

I believe that there is no way to do that with DSL. The reason is that we never go back to the template for lifecycle information after the data stream is created. Unlike the new settings/mappings stuff, we don't merge the template and data stream lifecycle information dynamically when we need it -- we just read what is on the data stream. So as it stands now, if you want to revert the lifecycle to be what is in the template, you have to explicitly read the template and set that value on your data stream.

[klacabane]

Thanks! Would have loved to avoid the extra step :) I think the behavior is fine even if by doing that we have to update the DSL: while a template can define a lifecycle, the data_retention is picked up by the stream at creation time and write indices won't pick up any updated retention but stick to the one effective at creation time. If the template is updated to use ILM, new write indices will be configured with the ILM so in that sense it follows the template as long as we make sure to not leave _settings override in place

[flash1293]

Agree, I think we should handle that on the Kibana side. For the user I think it's nice to be able to "remove" the streams layer in any case and go back to "inherit", even if it's not 100% what happens on the Elasticsearch layer. Keeps up the mental model of a layer on top of "raw" Elasticsearch you can add, but also take away.

We should definitely make sure to properly test this, seems like an easy one to break in subtle ways otherwise.