index_closed_exception when querying with _stats?expand_wildcards=open ; ignore_unavailable not documented

[tomasbanet]

Elasticsearch Version

7.16.1

Installed Plugins

repository-gcs

Java Version

bundled

OS Version

Darwin Kernel Version 20.6.0

Problem Description

Two issues related to hitting the _stats endpoint when some indices are closed.

Firstly simply calling /_stats is expected to fail when some indices are closed, as mentioned in https://github.com/elastic/elasticsearch/issues/15744. We need to explicitly set the param expand_wildcards=open. However, this option does not seem to work:

➜  curl "localhost:9200/_stats?expand_wildcards=open"
{"error":{"root_cause":[{"type":"index_closed_exception","reason":"closed","index_uuid":"<uuid>","index":"<index>"}],"type":"index_closed_exception","reason":"closed","index_uuid":"<uuid>","index":"<index>"},"status":400}

Instead we can try the ignore_unavailable flag, which does work:

➜  curl "localhost:9200/_stats?ignore_unavailable=true"
<stats with no error>

However this flag does not seem to be part of the Go API: https://pkg.go.dev/github.com/elastic/go-elasticsearch/[email protected]/esapi#IndicesStats. It's also not documented in the elastic API docs: https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-stats.html

Steps to Reproduce

1. Close some indices.
2. Call the /_stats endpoint with the flag expand_wildcards=open.
3. Observe an index_closed_exception error response.
4. Call /_stats with the flag ignore_unavailable=true.
5. Observe a successful response.

Logs (if relevant)

No response

[tomasbanet]

I have also tried to query with curl "localhost:9200/_stats?forbid_closed_indices=true" but that also returns an index_closed_exception

[astefan]

@tomasbanet it's not clear to me what is the issue here about. Please, can you clarify:

• is the go-elasticsearch client missing ignore_unavailable option the culprit?
• the fact that _stats accepts ignore_unavailable but the option is not documented is the issue?

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[tomasbanet]

Hi @astefan,

* is the `go-elasticsearch` client missing `ignore_unavailable` option the culprit?
* the fact that `_stats` accepts `ignore_unavailable` but the option is not documented is the issue?

Yes exactly, these two are separate issues that I have identified that relate to working around the index_closed_exception error on the _stats endpoint.

Plus, there is a third issue - /_stats?expand_wildcards=open gives the error index_closed_exception if there are closed indices. I am not sure if this is expected. Should closed indices be completely ignored when this flag is set?

[chandlermelton]

I can reproduce this on 7.16.3 when the closed index is part of an alias that would be caught by the query.

[astefan]

@tomasbanet for the go client, a separate issue needs to be opened in https://github.com/elastic/go-elasticsearch. And the issues around _stats in ES can stay in this issue. Thank you.

[DaveCTurner]

I have also tried to query with curl "localhost:9200/_stats?forbid_closed_indices=true" but that also returns an index_closed_exception

I think this is expected at least, if you want this API to permit closed indices (i.e. not to forbid them) then you should set ?forbid_closed_indices=false.

[synhershko]

@DaveCTurner this is very awkward to be honest, "forbidding" should not throw, just either ignore closed indices or report them as closed. It's a monitoring endpoint, I see no reason for it to throw?

Also the docs are not in-line with this behavior (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html):

forbid_closed_indices
(Optional, Boolean) If true, statistics are not collected from closed indices. Defaults to true.

This is also a regression, since this was not throwing before 7.3