[Backport 2.x] [Segment Replication] Backport PR's : #3525 #3533 #3540 #3943 #3963 From main branch

[Rishikesh1159]

Description

This PR backports following 5 PR's from main branch:

1. #3525.
2. #3533.
3. #3540.
4. #3943
5. #3963

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/1639/
• CommitID: d979fd2bf25af51d15acfacb1e99afa39dfdbcd0

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS :white_check_mark:
• URL: https://build.ci.opensearch.org/job/gradle-check/1647/
• CommitID: 4b60a6805505491b322dcb193487d8f9fae8be0a

[codecov-commenter]

Codecov Report

Merging #4181 (7f6da3f) into 2.x (7062283) will increase coverage by 0.01%. The diff coverage is 76.64%.

@@             Coverage Diff              @@
##                2.x    #4181      +/-   ##
============================================
+ Coverage     70.45%   70.46%   +0.01%     
- Complexity    56690    56769      +79     
============================================
  Files          4563     4569       +6     
  Lines        273024   273348     +324     
  Branches      40059    40083      +24     
============================================
+ Hits         192364   192628     +264     
- Misses        64521    64524       +3     
- Partials      16139    16196      +57     

Impacted Files	Coverage Δ
.../main/java/org/opensearch/index/engine/Engine.java	73.70% <ø> (+0.48%)	:arrow_up:
...va/org/opensearch/index/engine/InternalEngine.java	74.17% <ø> (+0.88%)	:arrow_up:
...rg/opensearch/indices/recovery/RecoveryTarget.java	70.80% <ø> (-0.03%)	:arrow_down:
...search/indices/recovery/RecoveryTargetHandler.java	100.00% <ø> (ø)
...rch/indices/recovery/RetryableTransportClient.java	81.81% <ø> (ø)
.../replication/checkpoint/ReplicationCheckpoint.java	42.42% <0.00%> (+6.06%)	:arrow_up:
...ckpoint/SegmentReplicationCheckpointPublisher.java	100.00% <ø> (ø)
...replication/common/ReplicationFailedException.java	12.50% <12.50%> (ø)
...s/replication/SegmentReplicationSourceService.java	56.81% <44.00%> (-24.27%)	:arrow_down:
...s/replication/SegmentReplicationTargetService.java	71.92% <60.00%> (-6.12%)	:arrow_down:
... and 498 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/1663/
• CommitID: 8c5753b9ef8379219ba36ae43167707c14a46069

[Rishikesh1159]

Flaky test:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.OngoingSegmentReplicationsTests.testMultipleReplicasUseSameCheckpoint" -Dtests.seed=A0971E691B2879E0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Asia/Kolkata -Druntime.java=17
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([A0971E691B2879E0:A4046FDAE5B7632]:0)
        at org.opensearch.index.shard.IndexShard.getProcessedLocalCheckpoint(IndexShard.java:2711)
        at org.opensearch.indices.replication.OngoingSegmentReplicationsTests.setUp(OngoingSegmentReplicationsTests.java:73)

Actually this is not flaky test. this happened due to changes in recent commit. Will fix this in next commit

[dreamer-89]

Flaky test:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.OngoingSegmentReplicationsTests.testMultipleReplicasUseSameCheckpoint" -Dtests.seed=A0971E691B2879E0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Asia/Kolkata -Druntime.java=17
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([A0971E691B2879E0:A4046FDAE5B7632]:0)
        at org.opensearch.index.shard.IndexShard.getProcessedLocalCheckpoint(IndexShard.java:2711)
        at org.opensearch.indices.replication.OngoingSegmentReplicationsTests.setUp(OngoingSegmentReplicationsTests.java:73)

Another appearance in https://github.com/opensearch-project/OpenSearch/pull/4041. Created https://github.com/opensearch-project/OpenSearch/issues/4184 to track it as it is failing more oftenly on CI.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/1664/
• CommitID: 8c5753b9ef8379219ba36ae43167707c14a46069

[Rishikesh1159]

Flaky test:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.OngoingSegmentReplicationsTests.testMultipleReplicasUseSameCheckpoint" -Dtests.seed=A0971E691B2879E0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Asia/Kolkata -Druntime.java=17
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([A0971E691B2879E0:A4046FDAE5B7632]:0)
        at org.opensearch.index.shard.IndexShard.getProcessedLocalCheckpoint(IndexShard.java:2711)
        at org.opensearch.indices.replication.OngoingSegmentReplicationsTests.setUp(OngoingSegmentReplicationsTests.java:73)

Another appearance in #4041. Created #4184 to track it as it is failing more oftenly on CI.

Sorry @dreamer-89 , I thought this is a flaky test but not. This happened because of adding "assert statement" back that we discussed in above comments. I will remove the assert to fix this

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS :white_check_mark:
• URL: https://build.ci.opensearch.org/job/gradle-check/1667/
• CommitID: 7f6da3f2856d5a877bbfe924c97720619d4f463d