Roman Levitsky

Here is the related master node log excerpt: `[2024-04-02T17:12:40,341][WARN ][o.o.r.RepositoriesService] [v4-master.company.com] [searchable] failed to finish repository verification org.opensearch.repositories.RepositoryVerificationException: [searchable] cannot delete test data at at org.opensearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:1986) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.repositories.RepositoriesService$3.lambda$doRun$1(RepositoriesService.java:383) [opensearch-2.12.0.jar:2.12.0]...

To collect more information, I've set `logger.level: DEBUG` at all four master-eligible nodes and restated them. And now my repository got successfully verified... UPD: few minutes later, it failed to...

I wrote a simple Python program to check the contents of the snapshot directory every 0.01 seconds. Here is it's output when search node stopped and repo verified successfully: [dir-ok.txt](https://github.com/opensearch-project/OpenSearch/files/14857149/dir-ok.txt)...

In case somebody interested, I managed to conquer this issue. Our cluster is running Oracle Linux Server 8.9 with ol8_UEKR6 kernel (5.4.17). New node is running Oracle Linux Server 8.9...

I was unable to identify the exact cause. What I've noticed that with 5.4 kernel, we see some time discrepancy (sometimes up to 2 seconds) between system clock and hardware...

Thank you @shwetathareja for your reply! Here are clarifications: 1. Yes we are running non-dedicated manager cluster, we have four nodes running both data and master-eligible nodes and two coordinating...

Re heap dump, where should we collect it and when? Right now, we see nothing at data nodes. Coordinating nodes sometimes telling something like `[INFO ][o.o.i.b.HierarchyCircuitBreakerService] [v484-coordinator.company.com] attempting to trigger...

Thank you Andriy for your reply. I've searched the https://github.com/opensearch-project/OpenSearch and was unable to find appropriate Dockerfile. Could you please point me to the right one?

I am unable to build OpenSearch image yet. Moreover, Dockerfile ( https://github.com/opensearch-project/opensearch-build/blob/main/docker/release/dockerfiles/opensearch.al2.dockerfile ) is telling that > This dockerfile generates an AmazonLinux-based image containing an OpenSearch installation (1.x Only). >...

Thank you Peter, However, neither I am a Java programmer nor a Docker enthusiast, and that _"inherit" from 2.12 image and install/replace JDK version to run with"_ doesn't seem to...