try to reproduce the issue when dclag increases after a fail back, which is a follow up step after the initial fail over

Replace this text with your description here...

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

[ ] The PR has a description, explaining both the problem and the solution.
[ ] The description mentions which forms of testing were done and the testing seems reasonable.
[ ] Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

[ ] This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
[ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

Jun 15 '23 22:06 flowguru

[root@haofu-dev-okteto-6f5fdf458d-g8hbl test]# grep DCLag *
grep: simfdb: Is a directory
trace.0.0.0.0.19213.1686869786.LjCPxf.0.1.xml:<Event Severity="10" Time="85.775382" DateTime="2023-06-15T22:56:28Z" Type="DCLag" Machine="0.0.0.0:0" ID="0000000000000000" Primary="272328991" Remote="272442129" ThreadID="3317230561354355491" LogGroup="default" />
trace.0.0.0.0.19213.1686869786.LjCPxf.0.1.xml:<Event Severity="10" Time="94.813886" DateTime="2023-06-15T22:56:29Z" Type="DCLag" Machine="0.0.0.0:0" ID="0000000000000000" Primary="280039361" Remote="280039361" ThreadID="3317230561354355491" LogGroup="default" />
trace.0.0.0.0.19213.1686869786.LjCPxf.0.1.xml:<Event Severity="10" Time="155.824928" DateTime="2023-06-15T22:56:30Z" Type="DCLag" Machine="0.0.0.0:0" ID="0000000000000000" Primary="340576319" Remote="340576319" ThreadID="3317230561354355491" LogGroup="default" />
trace.0.0.0.0.19213.1686869786.LjCPxf.0.1.xml:<Event Severity="10" Time="157.855053" DateTime="2023-06-15T22:56:30Z" Type="DCLag" Machine="0.0.0.0:0" ID="0000000000000000" Primary="342441521" Remote="342441521" ThreadID="3317230561354355491" LogGroup="default" />
trace.0.0.0.0.19213.1686869786.LjCPxf.0.1.xml:<Event Severity="10" Time="159.879499" DateTime="2023-06-15T22:56:30Z" Type="DCLag" Machine="0.0.0.0:0" ID="0000000000000000" Primary="342969986" Remote="342969986" ThreadID="3317230561354355491" LogGroup="default" />

with simulation test duration of 30, it does not seem to reproduce, will try to increase duration and see. @jzhou77

Jun 15 '23 22:06 flowguru

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 524bc47e6279399bf96eedf3929e2dd8506ebef8
Duration 0:20:45
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Jun 15 '23 23:06 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 524bc47e6279399bf96eedf3929e2dd8506ebef8
Duration 0:51:58
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Jun 15 '23 23:06 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 524bc47e6279399bf96eedf3929e2dd8506ebef8
Duration 0:53:30
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Jun 15 '23 23:06 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 524bc47e6279399bf96eedf3929e2dd8506ebef8
Duration 1:23:25
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

Jun 16 '23 00:06 foundationdb-ci

below is the result of the second commit

[root@haofu-dev-okteto-6f5fdf458d-g8hbl test]# grep DCLag * | grep -oP 'Primary="[0-9]*" Remote="[0-9]*"'
grep: simfdb: Is a directory
Primary="193141472" Remote="193141472"
Primary="193141472" Remote="193141472"
Primary="199475904" Remote="199475904"
Primary="201657440" Remote="201657440"
Primary="201657440" Remote="201657440"
Primary="203744463" Remote="203744463"
Primary="203744463" Remote="203744463"
Primary="205778162" Remote="205778162"
Primary="205778162" Remote="205778162"
Primary="207651303" Remote="207651303"
Primary="207651303" Remote="207651303"
Primary="209491658" Remote="209491658"
Primary="209491658" Remote="209491658"
Primary="211657493" Remote="211657493"
Primary="211657493" Remote="211657493"
Primary="213470513" Remote="213470513"
Primary="213470513" Remote="213470513"
Primary="215624715" Remote="215624715"
Primary="217786221" Remote="217786221"
Primary="217786221" Remote="217786221"
Primary="219505719" Remote="219505719"
Primary="219505719" Remote="219505719"
Primary="221476655" Remote="221476655"
Primary="221476655" Remote="221476655"
Primary="223456441" Remote="223456441"
Primary="223456441" Remote="223456441"
Primary="225523173" Remote="225523173"
Primary="227449407" Remote="227449407"
Primary="917617327" Remote="917617327"
Primary="926538667" Remote="926538667"
Primary="928598483" Remote="928598483"
Primary="930470513" Remote="930470513"
Primary="1243888188" Remote="1243888188"
Primary="1251535520" Remote="1251535520"
Primary="1253498347" Remote="1253498347"
Primary="1255624780" Remote="1255624780"

it seems DCLag from ConsistencyCheck does not have a gap, next step i will see how to check the DCLag during the test.

Jun 16 '23 19:06 flowguru

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 6c67475735598b3d25fcd45a4186bda1e3af5966
Duration 0:19:49
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Jun 16 '23 19:06 foundationdb-ci

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 6c67475735598b3d25fcd45a4186bda1e3af5966
Duration 0:43:37
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Jun 16 '23 20:06 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 6c67475735598b3d25fcd45a4186bda1e3af5966
Duration 1:08:25
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Jun 16 '23 20:06 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 6c67475735598b3d25fcd45a4186bda1e3af5966
Duration 1:13:58
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Jun 16 '23 20:06 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 6c67475735598b3d25fcd45a4186bda1e3af5966
Duration 1:23:46
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

Jun 16 '23 20:06 foundationdb-ci