Bad request thrown when alias points to a closed index and security enabled

[bmcconaghy]

Elasticsearch version (bin/elasticsearch --version): 6.3.0 and earlier

Plugins installed: none

JVM version (java -version): java version "10" 2018-03-20 Java(TM) SE Runtime Environment 18.3 (build 10+46) Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10+46, mixed mode) OS version (uname -a if on a Unix-like system): Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64 Description of the problem including expected versus actual behavior: _search/* throws a 400 bad request when an alias points at a closed index and security is enabled. I expect this not to happen. Steps to reproduce:

1. Start Elasticsearch (with security enabled) Note that I can only reproduce it in a cluster with security enabled; if disabled, I cannot reproduce it.
2. Create 2 test indices and an alias: POST test1/test1 { "test":"test" }

POST test2/test1 { "test":"test" }

POST /_aliases { "actions" : [ { "add" : { "index" : "test1", "alias" : "alias1" } }, { "add" : { "index" : "test2", "alias" : "alias1" } } ] } 3. do a search POST _search { "size":0, "aggs": { "indices":{"terms":{"field":"_index","size":200}}}}

response: { "error": { "root_cause": [ { "type": "index_closed_exception", "reason": "closed", "index_uuid": "OO462ue0QQCUQ_lS1SvHTA", "index": "test1" } ], "type": "index_closed_exception", "reason": "closed", "index_uuid": "OO462ue0QQCUQ_lS1SvHTA", "index": "test1" }, "status": 400 }

This issue causes an error in Kibana: https://github.com/elastic/kibana/issues/20920

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

This is essentially the same as #29948. A little over a year ago, I proposed a change that had the following description:

This change resolves a long standing TODO in the security indices and aliases resolution code when determining if an alias is visible. Previously, the code only checked if the indices options had the ignore aliases option set. In this change, each index referenced by the alias has its current state compared to the values of expand wildcards open and expand wildcards closed from the indices options. If one of the indices has a conflict with the values from the indices options then the alias is not considered visible.

@javanna reviewed the change and raised issues with this change. This is just an excerpt of what was said:

Elasticsearch was changed a long while ago to look at state of indices backing aliases. I think that we look at expand_wildcards only when the alias is matched through a wildcard expression though. What happens in that case is that we take out the indices that don't have the expected state, so we end up potentially resolving an alias to a subset of the indices that it points to. We can do that because we are expanding to concrete indices hence e.g. an alias that points to index-closed and index-open will be expanded to index-open only. But if you refer directly to the alias, it will be expanded to all the indices that are backing it regardless of their state.

In security, things are more complicated as we don't want to resolve aliases to concrete indices, they have to stay as they are, which makes it impossible to do what Elasticsearch does (filtering some of the indices out based on their state). I am not sure that looking at expand_wildcards all the time (and not only for aliases matched by wildcard expressions) is good, though the Elasticsearch behaviour described above is debatable too.

The reason why we need to keep aliases as they are is document level security, as users may have roles configured against aliases but not against their corresponding concrete indices. Given that we have a new way of doing document level security at this point though, we may want to change this, and I think that would be the definitive solution.

Ultimately, we ended up saying that the short-term fix should be #29952. That is to create a custom instance of indices options that is lenient (meaning ignore_unavailable set to true and allow_no_indices set to true) but that forbids closed indices. Longer term we wanted to make some under the cover changes https://github.com/elastic/elasticsearch/issues/29874 that require permissions on indices even when accessed using an alias.

[bmcconaghy]

What's the disposition of this? It causes an unrecoverable issue in Kibana when users have aliases pointed at closed indices.

[jaymode]

@bmcconaghy is this something Kibana can handle? We do not have a clear way forward on our end and it has been a longstanding issue

[bmcconaghy]

@jaymode Not really, no. We can just swallow the exception, but that means that you can't create an index pattern at all when you've got this situation. If you already had an index pattern that matches the alias before this situation came up, you would get fatal errors in discover and visualize. Again, we could swallow the exception, but then you would be shown no data.

[jaymode]

@javanna do you think adding forbid_closed_indices as a valid request parameter is an option?

[bmcconaghy]

Would that mean that Kibana would have to pass that with every request? That would be a pain to trace through all the places we would need to add it.

[bmcconaghy]

Or reading this again, maybe it is the reverse, that you would have to pass that to get the current behavior?

[javanna]

@javanna do you think adding forbid_closed_indices as a valid request parameter is an option?

forbid_closed_indices is about how API behave and should not be a configurable option. I think that by exposing it we would make things worse. The plan should rather be to try and separate the indices options between internal ones and external ones so internal ones don't ever end up being exposed.

One thing I don't understand is why this comes up now, as this issue existed for a very long time. Not saying it's good, but it's a known limitation of our security model that allows for document based security through filtered aliases. I will follow-up with Jay to try and see if there's something we can actually do to address it.

[jaymode]

@javanna and I talked and he reminded me where the real issue lies. Security allows authorization on aliases even if the user is not granted permission to an underlying index. This conflicts with how the default elasticsearch wildcard resolution works which resolves the alias to only the open indices, but security keeps the alias in the request. Essentially there is no good way forward until we remove support for authorizing on aliases #29874

[not-napoleon]

This issue also impacts the /_cat/shards API call, which gets used a lot in diagnosing issues. Came up while helping support, and I was able to repro locally against master. Same setup - create two indices, set up an alias to point to both, close one, try to run /_cat/shards. Whole call will fail with the index closed exception. I can provide a gist to repro from if that helps.

[andrewthad]

I just ran into the same issue that @not-napoleon described. The fact that closing indices breaks /_cat/shards is a problem for us, and it means that we essentially cannot ever close indices.

[PhaedrusTheGreek]

This also seems to affect a number of other API calls including:

_stats
_segments
_shards

[cybersecdiva]

I have observed this in an exact scenario for v7.7 as well with _statsAPI call.

[cubixx]

I also have the same issue. As I have closed indices referenced by aliases, I cannot use the _cat/shards API. Is there a workaround we can use?

[conicl]

I just ran into the same issue and I am trying some solutions. I can understand we need to keep aliases to cover the following case:

Users may have roles configured against aliases but not against their corresponding concrete indices.

However, can we separate the indices list used to buildIndicesAccessControl(...) from the part used to replace the indices for a IndicesRequest.Replaceable? When expand wildcards, if permission detection can be passed, the index is appended to the list used to buildIndicesAccessControl(...). Also, this index, if it is an aliases, we can get the backing index and filter it according to IndicesOptions and add it to a separate list which is use to replaces the indices for the Request at last.

Or, I think we can add and IndicesOptions : expandWildcardsAlias which control if a Index is visible when it is resolved from wildcards.

[thateriksson]

Is this issue a problem on 7+ versions too I have a self-managed client who is complaining about this and they are on version 7.11

[lucabelluccini]

The problem also affects Data Streams. At least in 7.17.3 I can reproduce with:

# GET _data_stream/logs-endpoint.events.registry-default
{
  "data_streams" : [
    {
      "name" : "logs-endpoint.events.registry-default",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-logs-endpoint.events.registry-default-2021.08.11-000001",
          "index_uuid" : "_9LUd4tKS6m5IDj3JE5DYQ"
        },
        {
          "index_name" : ".ds-logs-endpoint.events.registry-default-2022.02.07-000007",
          "index_uuid" : "QPtaaA5PQ1ymOd6jap6iVw"
        },
        {
          "index_name" : ".ds-logs-endpoint.events.registry-default-2022.03.09-000013",
          "index_uuid" : "LD9C9oDESAaBDJf9o1bBBQ"
        },
        {
          "index_name" : ".ds-logs-endpoint.events.registry-default-2022.04.08-000014",
          "index_uuid" : "qVDp5vfyQMS0OZ4dFxajtw"
        },
        {
          "index_name" : ".ds-logs-endpoint.events.registry-default-2022.05.08-000015",
          "index_uuid" : "fTbLJZEmTauf4iZmXRU77g"
        }
      ],
...

# POST .ds-logs-endpoint.events.registry-default-2021.08.11-000001/_close

# POST logs-endpoint.events.registry-*/_field_caps?fields=*&ignore_unavailable=true
{
  "error" : {
    "root_cause" : [
      {
        "type" : "cluster_block_exception",
        "reason" : "index [.ds-logs-endpoint.events.registry-default-2021.08.11-000001] blocked by: [FORBIDDEN/4/index closed];"
      }
    ],
    "type" : "cluster_block_exception",
    "reason" : "index [.ds-logs-endpoint.events.registry-default-2021.08.11-000001] blocked by: [FORBIDDEN/4/index closed];"
  },
  "status" : 403
}

# GET logs-endpoint.events.registry-default/_search?ignore_unavailable=true
Works fine

# GET logs-endpoint.events.registry-default/_search?ignore_unavailable=false
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_closed_exception",
        "reason" : "closed",
        "index_uuid" : "_9LUd4tKS6m5IDj3JE5DYQ",
        "index" : ".ds-logs-endpoint.events.registry-default-2021.08.11-000001"
      }
    ],
    "type" : "index_closed_exception",
    "reason" : "closed",
    "index_uuid" : "_9LUd4tKS6m5IDj3JE5DYQ",
    "index" : ".ds-logs-endpoint.events.registry-default-2021.08.11-000001"
  },
  "status" : 400
}

Note:

• Field Caps API fails even if we pass &ignore_unavailable=true
• Search with &ignore_unavailable=true works fine

[albertzaharovits]

Core interprets explicitly mentioned aliases differently from the ones covered by wildcards (for the former, in some APIs, it throws an error when they point to closed indices). But the Security filter replaces wildcards with the authorized resource names (aliases, indices, and datastreams). Hence Core cannot subsequently determine which aliases were mentioned directly from those which Security put in place to replace wildcards.

Security can technically plug into the Core algorithm, such that the plugged-in Core algorithm can deal with wildcards. But that doesn't mean that the Security filter can pass on the wildcard expansion. The wildcard expansion is still needed for the DLS/FLS features disable in the request interceptors as well as for auditing.

The nub is that we need to do some for of wildcard expansion in the Security filter, but still plug into the Core's algorithm to let it know for which alias names it should error if they point to closed indices.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[n1v0lg]

@albertzaharovits @slobodanadamovic none of the recent work around authz improvement address this right? Am I correct in saying that this remains an issue and that we currently haven't prioritized working on it?

[MarianaMartinYuste2]

Hello @n1v0lg @albertzaharovits @slobodanadamovic Would you any update on this, please? Thanks

[albertzaharovits]

Yes, none of the recent authz improvements has addressed this yet. This issue is still standing as described.

Le mer. 15 mars 2023, 10:42, Mariana @.***> a écrit :

Hello @n1v0lg https://github.com/n1v0lg @albertzaharovits https://github.com/albertzaharovits @slobodanadamovic https://github.com/slobodanadamovic Would you any update on this, please? Thanks

— Reply to this email directly, view it on GitHub https://github.com/elastic/elasticsearch/issues/32238#issuecomment-1470134313, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABC3KZFDTGSSRBYE5ZWNKB3W4HIOHANCNFSM4FLAC5GQ . You are receiving this because you were mentioned.Message ID: @.***>