Changing settings on `.reporting*` indices is not allowed, but we encourage users to do it

[masseyke]

Elasticsearch Version

8.0.0 or higher

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

The kibana plugin creates .reporting* system indices (https://github.com/elastic/elasticsearch/blob/main/modules/kibana/src/main/java/org/elasticsearch/kibana/KibanaPlugin.java#L34). Those indices are managed by an ILM policy called kibana-reporting (https://www.elastic.co/guide/en/kibana/current/reporting-getting-started.html). Kibana allows users to modify the index priority on the page for editing an ILM policy, and that is stored as an index setting. We do not allow index settings to be changed on system indices, even running as a user with a role with the all privilege and allow_restricted_indices set to true. Maybe we either need to whitelist the .reporting* indices to allow changing settings, or make the .reporting* indices not system indices. I'm not really sure, so turning it over to the team that maintains this.

Below is an example stack trace when running as a user with the superuser role, but it's fairly similar even with more elevated privileges:

[instance-0000000019] policy [kibana-reporting] for index [.reporting-2019.09.01] failed on step [{"phase":"warm","action":"set_priority","name":"set_priority"}]. Moving to ERROR step org.elasticsearch.ElasticsearchSecurityException: action [indices:admin/settings/update] is unauthorized for user [2139699331] with roles [superuser] on restricted indices [.reporting-2019.09.01], this action is granted by the index privileges [manage,all]
     at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:36) ~[x-pack-core-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:906) ~[x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService.actionDenied(AuthorizationService.java:883) ~[x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:966) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:950) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:909) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:378) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:993) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:370) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:441) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:378) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$2(AuthorizationService.java:263) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:139) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:185) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:47) [x-pack-core-8.3.3.jar:?]
     at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:235) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
     at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
     at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:200) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:175) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:136) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:265) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) [x-pack-security-8.3.3.jar:?]
     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:165) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:311) [x-pack-core-8.3.3.jar:?]
     at org.elasticsearch.xpack.ilm.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:55) [x-pack-ilm-8.3.3.jar:8.3.3]
     at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1271) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.updateSettings(AbstractClient.java:1566) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.xpack.core.ilm.UpdateSettingsStep.performAction(UpdateSettingsStep.java:50) [x-pack-core-8.3.3.jar:?]
     at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.maybeRunAsyncAction(IndexLifecycleRunner.java:385) [x-pack-ilm-8.3.3.jar:8.3.3]
     at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.clusterStateProcessed(IndexLifecycleRunner.java:329) [x-pack-ilm-8.3.3.jar:8.3.3]
     at org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor$1.onResponse(MasterService.java:486) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor$1.onResponse(MasterService.java:483) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.cluster.service.MasterService$ExecutionResult.onPublishSuccess(MasterService.java:830) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.cluster.service.MasterService$1.onResponse(MasterService.java:325) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.cluster.service.MasterService$1.onResponse(MasterService.java:320) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:473) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:413) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:304) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:156) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:110) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:148) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:260) [elasticsearch-8.3.3.jar:?]
     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:223) [elasticsearch-8.3.3.jar:?]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
     at java.lang.Thread.run(Thread.java:833) [?:?]

Steps to Reproduce

• In kibana edit the kibana-reporting policy, add the warm phase, and set a priority.
• Create any report. For example in Discover -> Share -> CSV Report. Verify we get .reporting-{{date}} index created
• Change ILM poll interval or wait
• You'll see an error in logs like

[instance-0000000000] policy [kibana-reporting] for index [.reporting-2022-08-21] failed on step [{"phase":"warm","action":"set_priority","name":"set_priority"}]. Moving to ERROR step

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[holy-moly-555]

Any progress or news on this issue? Our Elasticsearch-Log gets spammed with messages like above...