Unable to update index pattern

[rlevytskyi]

Describe the bug At some point, our users started complaining they are unable to update their index patterns. Even I as cluster_admin unable to update them, getting this at kibana log:

{"type":"log","@timestamp":"2024-02-29T07:56:52Z","tags":["error","opensearch","data"],"pid":1,"message":"[security_exception]: Update is not supported when FLS or DLS or Fieldmasking is activated"} {"type":"error","@timestamp":"2024-02-29T07:56:52Z","tags":[],"pid":1,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:127:19)\n at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:83:19)\n at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:79:17)\n at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:175:34)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:140:50)\n at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://logs.company.com/api/saved_objects/index-pattern/178bc3f0-0d28-11ea-860b-8fae5a22bf63","message":"Internal Server Error"} {"type":"response","@timestamp":"2024-02-29T07:56:52Z","tags":[],"pid":1,"method":"put","statusCode":500,"req":{"url":"/api/saved_objects/index-pattern/178bc3f0-0d28-11ea-860b-8fae5a22bf63","method":"put","headers":{"connection":"upgrade","host":"logs.company.com","x-forwarded-for":"10.139.128.40","content-length":"406094","sec-ch-ua":""Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36","osd-version":"2.12.0","sec-ch-ua-platform":""Linux"","accept":"/","origin":"https://logs.company.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://logs.company.com/app/management/opensearch-dashboards/indexPatterns/patterns/178bc3f0-0d28-11ea-860b-8fae5a22bf63","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9","securitytenant":""},"remoteAddress":"127.0.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36","referer":"https://logs.company.com/app/management/opensearch-dashboards/indexPatterns/patterns/178bc3f0-0d28-11ea-860b-8fae5a22bf63"},"res":{"statusCode":500,"responseTime":65,"contentLength":9},"message":"PUT /api/saved_objects/index-pattern/178bc3f0-0d28-11ea-860b-8fae5a22bf63 500 65ms - 9.0B"}

And this at coordinating node log:

[2024-02-29T07:56:52,087][WARN ][r.suppressed ] [v48-coordinator.company.com] path: /.kibana/_update/index-pattern%3A178bc3f0-0d28-11ea-860b-8fae5a22bf63, params: {if_seq_no=925, if_primary_term=1, refresh=wait_for, index=.kibana, _source_includes=namespace,namespaces,originId, id=index-pattern:178bc3f0-0d28-11ea-860b-8fae5a22bf63} org.opensearch.OpenSearchSecurityException: Update is not supported when FLS or DLS or Fieldmasking is activated at org.opensearch.security.configuration.DlsFlsValveImpl.invoke(DlsFlsValveImpl.java:260) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:390) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.12.0.jar:2.12.0] ... at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.106.Final.jar:4.1.106.Final] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

To Reproduce Steps to reproduce the behavior:

1. Use OpenSearch as usual with Keycloak SAML
2. At some point, note that some sysadmins don't have access to all indices as they should
3. Waste a lot of time to find that is the case when sysadmin is also a member of some Dev team
4. Waste a lot of time to find that there was a similar issue for user that has several roles with no solution but with workaround, https://forum.opensearch.org/t/how-is-dls-applied-when-user-has-multiple-roles/2946
5. Implement a workaround and make sysadmins able to browse all indices again
6. After some while, receive complain at index pattern update
7. Check yourself and see error as described above
8. Temporarily remove this from the SA role: "dls": "{ \"bool\": { \"must_not\": [ { \"match_phrase\": { \"host\": \"v000.company.com\" } } ] } }" and see index pattern can now be updated.
9. But now SA cannot see all indices again because of overlapping roles.

Expected behavior The initial reason is that roles got overlapped when one has empty DSL.

OpenSearch Version OpenDistro 1.0 through OpenSearch 2.12

Dashboards Version OpenDistro 1.0 through OpenSearch 2.12

Plugins

• Security plugin with Keycloak SAML

Screenshots Not applicable, see logs.

Host/Environment (please complete the following information):

• OS: Ubuntu 23.10
• Chrome (doesn't matter acutally)

Additional context Will be added upon request.