[flaky test] Several org.yb.loadtester.* test suites

[m-iancu]

Jira Link: DB-3388

Description

org.yb.loadtester.TestClusterWithHighLoadAndSlowSync.testClusterFullMoveWithHighLoadAndSlowSync

org.yb.loadtester.TestFullMoveWithHeartBeatDelay.testClusterFullMoveWithHeartbeatDelay

org.yb.loadtester.TestMasterLeaderDecommission.testMasterLeaderDecommission

org.yb.loadtester.TestClusterFullMove.testClusterFullMove

org.yb.loadtester.TestRF1Cluster.testDefaultTTLWithChangedRF

org.yb.loadtester.TestRF1Cluster.testRF1toRF3

Looks like they all significantly increased the failure rate recently -- regression was introduced in this range for all of them: https://github.com/yugabyte/yugabyte-db/compare/34ca65fd605a95173ed7ffabdf2ee14e86757585~...c1d676580933d34bd34fcbb7e86099dcd9e0eb99

From the detective diff results looks like this diff might be the cause: https://github.com/yugabyte/yugabyte-db/commit/9c8e831302822a18170ed5f7c5a4350fb4472f32

[bmatican]

This one seems to have been failing indeed for some time

https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&build_type=all&class=org.yb.loadtester.TestClusterWithHighLoadAndSlowSync&fail_tag=all&name=testClusterFullMoveWithHighLoadAndSlowSync&platform=linux

[tedyu]

There was a proposed change deadlock_detector.cc . However, it seems no current test goes through that code path.

Edit: I am looking closer at the tests since the test stability may have improved due to infra changes.

[bmatican]

Not showing up in top 20% failure rate anymore