Operator/index templates

[grcevski]

This PR adds support for /_index_template/ and /_component_template/ in file based settings.

Relates to https://github.com/elastic/elasticsearch/issues/89183

[elasticsearchmachine]

Hi @grcevski, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[andreidan]

@grcevski A final question that just dawned on me.

How are we handling existing clusters that might have component/composable templates that start with the chosen prefix for the reserved templates? Will they essentially become reserved when upgrading to a version that has the reserved templates feature?

[grcevski]

How are we handling existing clusters that might have component/composable templates that start with the chosen prefix for the reserved templates? Will they essentially become reserved when upgrading to a version that has the reserved templates feature?

I don't think it should be a problem, nothing should happen to the templates, but maybe I'm missing a case here.

Essentially we only add the prefix when we store the names of the templates in the reserved cluster state section of the metadata, the templates themselves get stored in the custom storage without the prefix. To store any reserved state we must have defined a template in the file settings, otherwise there's nothing that modifies the reserved sections of the cluster state metadata.

Here's what I think the potential situations might be:

1. We have an existing composable index template called composable_index_template:foo which uses the same prefix we intend to use for reservation in the metadata.

   • We want to add new reserved template foo through file based settings. We store composable_index_template:foo in the reserved section of the metadata and a template with name foo in the composable index templates metadata. There should be nothing that prevents this, we'll end up with two different composable index templates, one called foo another called composable_index_template:foo, but only foo is reserved. If one were to look at the full metadata dump, it might look confusing, but I think that's the extent of it.
   • Now we want to modify a template through the REST calls, say through TransportPutComponentTemplateAction, which defines the modified keys as Set.of(ReservedComposableIndexTemplateAction.reservedComponentName(request.name())), essentially prefixing the name with composable_index_template:.
     • If we wanted to modify foo through REST the modified keys we produce for conflict detection will be composable_index_template:foo, which will match the reserved keys metadata and we'll disallow the modification.
     • if we wanted to modify composable_index_template:foo through REST, the modified keys will produce a doubled prefix composable_index_template:composable_index_template:foo which won't match our reserved metadata and the REST call will be allowed.

2. We have an existing composable index template called foo. In this case the file based settings will overwrite it and put reserved metadata with composable_index_template:foo. This is the same behaviour as all other file based settings.

This was a very good call-out, I'm going to write some tests to see if I can break the logic and cause a conflict where there shouldn't be one if the user had already used our prefix in the name of a template.

[andreidan]

@grcevski thanks so much for the detailed explanation and extra tests 🚀 ❤️

Just toyed around a bit with upgrading an 8.4 cluster that had component_template:abc and composable template composable_template:logs_data_stream and indeed modifying and retrieving them through REST works like a champ after upgrade.