OpenSearch

OpenSearch copied to clipboard

[Remote Store] Permit backed futures to prevent timeouts during upload bursts

34

### Description During burst of uploads happening typically in finalize recovery in cases like shrink, split and force merge where lot of segments or large segments become available for upload,...

vikasvb90

[Feature Request] Field level statistics of lucene index files

4

### Is your feature request related to a problem? Please describe It seems impossible to get statistics such as disk consumption of individual lucene segment files per field. We do...

rishabhmaurya

[BUG] Cleanup unreferenced files when any operation failed due to disk full.

2

### Describe the bug As of now, we cleanup unreferenced files whenever last write is performed by [merge ](https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/engine/Engine.java#L1370)and it caused disk to get full and shard to fail. Incase...

RS146BIJAY

[Remote Store] Move blocking refreshes on writes off write thread pool

1

### Is your feature request related to a problem? Please describe With wait for refresh semantics write threadpool would be holding up a single lock for refresh till all upload...

Bukhtawar

[BUG] org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotRepository is flaky

1

### Describe the bug The test case `org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotRepository` is flaky: ``` org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotRepository java.lang.AssertionError: max seq. no. [1] does not match [0] at __randomizedtesting.SeedInfo.seed([5705DA441D4A7808]:0) at org.opensearch.index.engine.ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:202) at org.opensearch.index.engine.ReadOnlyEngine.ensureMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:189) at org.opensearch.index.engine.ReadOnlyEngine.(ReadOnlyEngine.java:140) at...

reta

[Remote Store] Cluster State Applier thread blocked on remote store operations

2

### Describe the bug On remote store clusters, we can see cluster state applier thread is blocked on remote store calls. In case when the calls to remote store takes...

gbbafna

[Remote Store] Add setting to limit primary shards per node for an index / all indices

2

### Is your feature request related to a problem? Please describe Today we have a setting `index.routing.allocation.total_shards_per_node` , which can limit maximum number of shards per index/across all indices on...

gbbafna

[Feature Request] Shallow Copy Snapshot Enhancements

3

### Is your feature request related to a problem? Please describe With full copy snapshots where we upload all the local segments data to snapshot repository, snapshot creation latency is...

harishbhakuni

[Remote Store] Update AWS SDK version to fix connection acquisition timeout issues

2

### Describe the bug We are seeing timeouts in connection acquisition from S3 client thread pool since we are hitting this AWS SDK bug: https://github.com/aws/aws-sdk-java-v2/pull/4066 . This is causing issues...

shourya035

[BUG] org.opensearch.remotestore.multipart.RemoteStoreMultipartIT is flaky

3

### Describe the bug The test case `org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testNoSearchIdleForAnyReplicaCount` is flaky: ``` org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testNoSearchIdleForAnyReplicaCount com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=12912, name=Thread-7889, state=RUNNABLE, group=TGRP-RemoteStoreMultipartIT] Caused by: org.apache.lucene.store.AlreadyClosedException: this Directory is closed...

reta