OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

add TestIssueLogging on flaky test - org.opensearch.remotestore.RemoteStoreRestoreIT.testRTSRestoreWithNoDataPostRefreshPrimaryReplicaDown

Open linuxpi opened this issue 9 months ago • 8 comments

Description

  • Adds test logging to debug flaky test.
  • Flaky Test org.opensearch.remotestore.RemoteStoreRestoreIT.testRTSRestoreWithNoDataPostRefreshPrimaryReplicaDown is not repro-able in local environment even after 5k iteration.
  • To be able to understand the root cause of flakyness, we need to get trace logging when this test fails.
  • Changes in the PR will ensure whenever this test fails in any PR gradle check, we get trace logging to be able to find the root cause and fix

Related Issues

Relates to https://github.com/opensearch-project/OpenSearch/issues/11085

Check List

  • [x] New functionality includes testing.
    • [x] All tests pass
  • [ ] ~New functionality has been documented.~
    • [ ] ~New functionality has javadoc added~
  • [x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • [x] Commits are signed per the DCO using --signoff
  • [ ] ~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~
  • [ ] ~Public documentation issue/PR created~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

linuxpi avatar May 01 '24 18:05 linuxpi

:x: Gradle check result for 6ba76904371cd744ff4b7d0a20373f1fd2bd92f8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar May 01 '24 18:05 github-actions[bot]

@linuxpi this links to an ES bug? Can you explain what you're trying to do please?

dblock avatar May 01 '24 19:05 dblock

@linuxpi this links to an ES bug? Can you explain what you're trying to do please?

@dblock this is related to a flaky bug - https://github.com/opensearch-project/OpenSearch/issues/11085 . We are enabling trace logging for test failures in PR builds to get to the root cause of the flakyness

linuxpi avatar May 02 '24 09:05 linuxpi

:x: Gradle check result for 2b9829673fdab3d2214a5ba30ce4e7f86b2c2cdc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar May 02 '24 10:05 github-actions[bot]

:x: Gradle check result for e3576561bdd54d7e3ef1fb9dfcf262f92ee23a74: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar May 02 '24 11:05 github-actions[bot]

:x: Gradle check result for bb3c70751b4b72f2ff788e7d66c332987a19370d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar May 02 '24 17:05 github-actions[bot]

:white_check_mark: Gradle check result for cbae09fa96d6d37bf08346d51bab9fe3faba55ca: SUCCESS

github-actions[bot] avatar May 03 '24 08:05 github-actions[bot]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 71.62%. Comparing base (b15cb0c) to head (e1466b5). Report is 264 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #13489      +/-   ##
============================================
+ Coverage     71.42%   71.62%   +0.20%     
- Complexity    59978    61105    +1127     
============================================
  Files          4985     5051      +66     
  Lines        282275   287119    +4844     
  Branches      40946    41602     +656     
============================================
+ Hits         201603   205658    +4055     
- Misses        63999    64467     +468     
- Partials      16673    16994     +321     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar May 03 '24 08:05 codecov[bot]

The link is to elastic/elasticsearch, it should be opensearch-project/OpenSearch

thanks @dblock for catching this. i thought you were referring to the link in description. I've corrected the link now

linuxpi avatar May 04 '24 11:05 linuxpi

:x: Gradle check result for e89dbdcdc4a93b86a7a8bac956560ad6c7f30d28: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar May 04 '24 11:05 github-actions[bot]

@linuxpi Noticed your empty commit to retrigger gradle check. You can do a git commit --amend and force push with -f to avoid that.

dblock avatar May 04 '24 13:05 dblock

:x: Gradle check result for f35bc18759f772002ef13d9e3f883d54c7bfab12: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar May 04 '24 13:05 github-actions[bot]

@linuxpi Noticed your empty commit to retrigger gradle check. You can do a git commit --amend and force push with -f to avoid that.

Thanks for the suggestion @dblock . Was trying to avoid force pushes but i guess just amending last commit might not be an issue

linuxpi avatar May 06 '24 05:05 linuxpi

:white_check_mark: Gradle check result for c4ff6de8bf44d66d427a5976c1debdbb57aabf93: SUCCESS

github-actions[bot] avatar May 06 '24 06:05 github-actions[bot]

:white_check_mark: Gradle check result for e1466b5a86c708a00ab20777caccaf9bdfee9517: SUCCESS

github-actions[bot] avatar May 06 '24 12:05 github-actions[bot]

@dblock can you check if the PR looks good now?

linuxpi avatar May 06 '24 15:05 linuxpi