Deleting an index during concurrent taking of more than one snapshot causes future restore to fail

[aYukiSekiguchi]

Describe the bug Elasticsearch 7.10.2 which OpenSearch forked has a known issue about snapshot and restore: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/release-notes-7.10.2.html#known-issues-7.10.2

If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail.

To Reproduce Steps to reproduce the behavior: I haven't reproduced the behavior, but I guess...

Delete index while the cluster is concurrently taking more than one snapshot. There is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail.

Expected behavior Concurrent snapshot and restore work.

Plugins I haven't reproduced the behavior, but I guess no plugin is needed.

Screenshots None

Host/Environment (please complete the following information): I haven't reproduced the behavior, but I guess all environment are affected.

Additional context I checked the patch in Elasticsearch and the same file in OpenSearch main branch. It looks like OpenSearch has the same issue.

[Bukhtawar]

This is specific to Amazon OpenSearch Service which has been well documented here https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-operations.html#version_7_10

[dblock]

Is there something we should/can do in OpenSearch itself? If not close this?

[reta]

@dblock the bare bone OpenSearch works just fine:

{
    "acknowledged": true,
    "persistent": {
        "snapshot": {
            "max_concurrent_operations": "1"
        }
    },
    "transient": {}
}

As per @Bukhtawar , it is specific to AWS OpenSearch offering: only limited settings are supported.

[aYukiSekiguchi]

I understand it is the problem of Amazon OpenSearch Service that the mitigation setting doesn't work.

However, my question is "Does OpenSearch have the know issue which Elasticsearch 7.10.2 has?"

To clarify, I quote the known issue from Elasticsearch 7.10.2 Release Note:

Snapshot and restore: If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail.

...

This issue is fixed in Elasticsearch versions 7.13.1 and later. It is not possible to repair a repository once it is affected by this issue, so you must restore the repository from a backup, or clear the repository by executing DELETE _snapshot//*, or move to a fresh repository. For more details, see #73456.

[reta]

@aYukiSekiguchi OpenSearch is a fork of Elasticsearch as of 7.10.2, so it is very likely that the issue is still present

[dblock]

@aYukiSekiguchi @Bukhtawar lets reopen an re-describe the original problem? It sounds like the root issue is that “If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail.” We should fix this. Please note that we cannot take non-AL2 code from ES.

[aYukiSekiguchi]

I updated the description and removed about the mitigation setting because it confused some people.

Note for future readers: The snapshot.max_concurrent_operations mitigation in Elasticsearch 7.10.2 Release Note should work on the bare bone OpenSearch. However, it looks like Amazon OpenSearch Service doesn't support the setting.

[xuezhou25]

Updating my thoughts on the root cause: Snapshot creation and deletion can not be proceed at the same time. After a snapshot deletion, actions will be triggered on in-progress snapshots, see: https://github.com/opensearch-project/OpenSearch/blob/658f7a63ed160d63f9117ef0227d2ad875e558b6/server/src/main/java/org/opensearch/snapshots/SnapshotsService.java#L2897-L2905

If there is a snapshot creation in the queue during the snapshot deletion, which means there are unfinished shard snapshots in UNASSIGNED_QUEUED state, there will be a step to check the current shard status in updatedSnapshotsInProgress from this line: https://github.com/opensearch-project/OpenSearch/blob/6f6e84ebc54af31a976f53af36a5c69d474a5140/server/src/main/java/org/opensearch/snapshots/SnapshotsService.java#L2957

When an index is deleted before the check, ShardSnapshotStatus will be put into MISSING state, which is taken as shard snapshot "completed". Then there is an incorrect use in updating snapshotEntries, in this line: https://github.com/opensearch-project/OpenSearch/blob/6f6e84ebc54af31a976f53af36a5c69d474a5140/server/src/main/java/org/opensearch/snapshots/SnapshotsService.java#L2968 The method withStartedShards assumes that the ShardSnapshotStatus is completed, see: https://github.com/opensearch-project/OpenSearch/blob/6f6e84ebc54af31a976f53af36a5c69d474a5140/server/src/main/java/org/opensearch/cluster/SnapshotsInProgress.java#L552-L556

It lends to the completed snapshots not being finalized, and this is the code for finalizing the completed snapshots: https://github.com/opensearch-project/OpenSearch/blob/6f6e84ebc54af31a976f53af36a5c69d474a5140/server/src/main/java/org/opensearch/snapshots/SnapshotsService.java#L2972-L2974

The code defect may cause state consistency issue during restoring the snapshot: the snapshot status is uncompleted, while the ShardSnapshotStatus is completed.

AVERTISSEMENT: Uncaught exception in thread: Thread[opensearch[node_t0][clusterManagerService#updateTask][T#1],5,TGRP-DedicatedClusterSnapshotRestoreIT]
java.lang.AssertionError: Completed state must imply all shards completed but saw state [STARTED] and shards [[test-index][0]=>ShardSnapshotStatus[state=MISSING, nodeId=null, reason=missing index, generation=null]]