foundationdb icon indicating copy to clipboard operation
foundationdb copied to clipboard

Add BitFlip fault injection workload

Open jzhou77 opened this issue 11 months ago • 36 comments

Add a bit flip for messages from TLog to SS. This either causes SS crash due to serialization error or inconsistent data on the SS.

Once #11255 is landed, this types of error should be detected, hopefully resolved by restarting processes.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

  • [ ] The PR has a description, explaining both the problem and the solution.
  • [ ] The description mentions which forms of testing were done and the testing seems reasonable.
  • [ ] Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

  • [ ] This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
  • [ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

jzhou77 avatar Mar 23 '24 05:03 jzhou77

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: c588f6969e6902fd25fffe6fe530ca7422accc26
  • Duration 0:20:01
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Mar 23 '24 05:03 foundationdb-ci

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: c588f6969e6902fd25fffe6fe530ca7422accc26
  • Duration 0:34:13
  • Result: :x: FAILED
  • Error: no matching artifact paths found
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Mar 23 '24 05:03 foundationdb-ci

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: c588f6969e6902fd25fffe6fe530ca7422accc26
  • Duration 0:45:37
  • Result: :x: FAILED
  • Error: no matching artifact paths found
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Mar 23 '24 05:03 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: c588f6969e6902fd25fffe6fe530ca7422accc26
  • Duration 0:47:36
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Mar 23 '24 05:03 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: c588f6969e6902fd25fffe6fe530ca7422accc26
  • Duration 0:48:55
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Mar 23 '24 05:03 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: c588f6969e6902fd25fffe6fe530ca7422accc26
  • Duration 0:50:01
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Mar 23 '24 05:03 foundationdb-ci

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 72057072e25830e4da593d90538c0f6257e35107
  • Duration 0:20:54
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 02 '24 23:04 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 72057072e25830e4da593d90538c0f6257e35107
  • Duration 0:51:29
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Apr 02 '24 23:04 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 72057072e25830e4da593d90538c0f6257e35107
  • Duration 0:54:01
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 02 '24 23:04 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 72057072e25830e4da593d90538c0f6257e35107
  • Duration 0:55:28
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 02 '24 23:04 foundationdb-ci

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 047bd9c54b9567ebfd485721b1a74064b74e55d3
  • Duration 0:33:23
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 08 '24 23:04 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 047bd9c54b9567ebfd485721b1a74064b74e55d3
  • Duration 0:59:43
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 00:04 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 047bd9c54b9567ebfd485721b1a74064b74e55d3
  • Duration 1:02:13
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Apr 09 '24 00:04 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 047bd9c54b9567ebfd485721b1a74064b74e55d3
  • Duration 1:11:35
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 00:04 foundationdb-ci

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 9abda6075b45109fadee4ca6061941435b622417
  • Duration 0:19:56
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 00:04 foundationdb-ci

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 9abda6075b45109fadee4ca6061941435b622417
  • Duration 0:32:58
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 00:04 foundationdb-ci

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 9abda6075b45109fadee4ca6061941435b622417
  • Duration 0:46:28
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 00:04 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 9abda6075b45109fadee4ca6061941435b622417
  • Duration 0:47:59
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 00:04 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 9abda6075b45109fadee4ca6061941435b622417
  • Duration 0:48:36
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 00:04 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 9abda6075b45109fadee4ca6061941435b622417
  • Duration 0:54:11
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Apr 09 '24 00:04 foundationdb-ci

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: bbc0a2278c2df55560a1bd82bc2ae8af3578516e
  • Duration 0:21:30
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 23:04 foundationdb-ci

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: bbc0a2278c2df55560a1bd82bc2ae8af3578516e
  • Duration 0:33:17
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 23:04 foundationdb-ci

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: bbc0a2278c2df55560a1bd82bc2ae8af3578516e
  • Duration 0:47:10
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 23:04 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: bbc0a2278c2df55560a1bd82bc2ae8af3578516e
  • Duration 0:48:49
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 23:04 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: bbc0a2278c2df55560a1bd82bc2ae8af3578516e
  • Duration 0:52:20
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 09 '24 23:04 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: bbc0a2278c2df55560a1bd82bc2ae8af3578516e
  • Duration 0:52:59
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Apr 09 '24 23:04 foundationdb-ci

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 84e9e03ca8121012dc81e76cc228e0c708f3c309
  • Duration 0:20:05
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 10 '24 17:04 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 84e9e03ca8121012dc81e76cc228e0c708f3c309
  • Duration 0:39:58
  • Result: :x: FAILED
  • Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/ycsb:${FDB_VERSION}-${COMMIT_SHA} --file Dockerfile --target ycsb .. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Apr 10 '24 17:04 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 84e9e03ca8121012dc81e76cc228e0c708f3c309
  • Duration 1:15:30
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 10 '24 18:04 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 84e9e03ca8121012dc81e76cc228e0c708f3c309
  • Duration 1:23:55
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Apr 10 '24 18:04 foundationdb-ci