cockroach
cockroach copied to clipboard
cockroachdb
roachtest: ssh_problem failed
roachtest.ssh_problem failed with artifacts on master @ 1b1c8da55be48c174b7b370b305f42622546209f:
test schemachange/random-load failed due to test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(test_impl.go:291).Fatal: output in run_125153.622625054_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
Jira issue: CRDB-20896
roachtest.ssh_problem failed with artifacts on master @ 0c1c3e7777b28a30ebe41428fb173f0156e8968c:
test schemachange/random-load failed due to test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(test_impl.go:291).Fatal: output in run_133500.318499865_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 2d926e68000df659f282d4e4477329867b9a3323:
test schemachange/random-load failed due to test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(test_impl.go:291).Fatal: output in run_140012.557459920_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255
(test_impl.go:314).Errorf: test timed out (0s)
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
Some notes about the failure above:
Error classifiction
The failure was classified as an SSH flake but, in reality, the test timed out. You can see it in the message above, as well as by checking the test logs: the workload ran for 10 hours.
To consider (cc @smg260):
- Improve the
test timed outmessage to properly report time spent in the test. This could be done by reusing the default timeout [1] or (even better) displaying the actual duration (for example, by exposingduration[2]). - Improving SSH-flake detection. It seems that when the test runner intentionally stops processes started by roachprod, there's a chance we'll get a 255 error. There's probably a better way to deal with this. One quick and dirty, short term solution would be to not mark a failure an SSH-flake if one of the errors indicates that the test timed out.
[1] https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/test_runner.go#L877 [2] https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/test_impl.go#L373-L375
Actual test failure (workload timeout)
The workload ran for 10h and never finished. We're passing --max-ops 5000, so it technically should terminate when we reach that number of operations (+ the number of concurrent workers which in this case is 20).
The logs generated by the workload are fairly large (280MB). If we grep for a field in the JSON printed when a worker performs its random number of ops, I think we are able to get a count of the actual number of schema change ops the workload ran:
% rg expectedExecErrors run_140012.557459920_n1_workload_run_schemachange.log | wc -l
5643
This looks suspicious to me. This number shouldn't be > 5019 (5000 maxOps + enough ops for the remaining 19 workers to realize they should stop after maxOps is reached).
@fqazi Thoughts? Anything to be done here?
Let me dig into this, it looks like we are some how kept generating ops, which is super weird.
@renatolabs It looks like we ran into an infinite loop inside randParentColumnForFkRelation. I'll get a patch out for it shortly
Thanks @renatolabs .
Improve the test timed out message to properly report time spent in the test. This could be done by reusing the default timeout [1] or (even better) displaying the actual duration (for example, by exposing duration [2]).
test.log shows the correct actual timeout of 10h (this is the default, and incidentally quite high)
23:59:09 test_runner.go:936: test timed out after 10h0m0s; check __stacks.log and CRDB logs for goroutine dumps
The 0s you see is reported at the time we finally call t.Errorf and it uses the timeout in the TestSpec, which defaults to 0. This should probably use the actual timeout as in the above print.
Improving SSH-flake detection. It seems that when the test runner intentionally stops processes started by roachprod, there's a chance we'll get a 255 error. There's probably a better way to deal with this. One quick and dirty, short term solution would be to not mark a failure an SSH-flake if one of the errors indicates that the test timed out.
Agreed. However this is a symptom of us delaying the call to t.Errorf on timeout until teardown as completed [1], [2]. When the nodes are stopped, the long running ssh command will fail, resulting in a 255 exit code, and the subsequent reporting of that error - but since we delayed reporting the real cause (timeout), this error becomes the primary and gets reported as such. We could certainly look at prioritising the time out error.
[1] - https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/test_runner.go#L933 [2] - https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/test_runner.go#L1051
There is a secondary bug here inside cockroache's DROP SCHEMA implementation too, which makes this worse. I'll band-aid the infinite loop first since a faster failure is better here.
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/bank-multitable/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank-multitable/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_104333.097775667_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.137 -n 10.142.0.45 -n 10.142.1.138 -n 10.142.1.127 -n 10.142.1.162 \
--test bank-multitable --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/bank-multitable/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank-multitable/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_105547.897432309_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.174 -n 10.142.0.224 -n 10.142.1.98 -n 10.142.1.172 -n 10.142.0.217 \
--test bank-multitable --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/bank/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_105853.056711258_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.166 -n 10.142.1.167 -n 10.142.1.163 -n 10.142.1.171 -n 10.142.1.168 \
--test bank --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/bank/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/bank/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_111113.812210581_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.116 -n 10.142.1.114 -n 10.142.1.125 -n 10.142.1.124 -n 10.142.1.126 \
--test bank --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/g2/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/g2/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_113040.476678420_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.194 -n 10.142.1.199 -n 10.142.1.193 -n 10.142.1.192 -n 10.142.1.198 \
--test g2 --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/g2/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/g2/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_113653.263800916_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.159 -n 10.142.1.157 -n 10.142.1.120 -n 10.142.1.158 -n 10.142.1.155 \
--test g2 --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/multi-register/majority-ring failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring/run_1
(test_impl.go:291).Fatal: output in run_120825.027668848_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.0.152 -n 10.142.0.91 -n 10.142.0.96 -n 10.142.0.145 -n 10.142.0.98 \
--test multi-register --nemesis majority-ring > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/multi-register/parts-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/parts-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_121117.835152541_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.0.95 -n 10.142.0.236 -n 10.142.0.110 -n 10.142.0.154 -n 10.142.0.112 \
--test multi-register --nemesis parts --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/multi-register/split failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/split/run_1
(test_impl.go:291).Fatal: output in run_121247.132219570_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.0.82 -n 10.142.0.86 -n 10.142.0.45 -n 10.142.0.89 -n 10.142.0.70 \
--test multi-register --nemesis split > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/monotonic/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/monotonic/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_115128.494652605_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.96 -n 10.142.0.48 -n 10.142.0.7 -n 10.142.0.9 -n 10.142.1.122 \
--test monotonic --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/multi-register/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_121653.485694934_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.97 -n 10.142.1.99 -n 10.142.1.91 -n 10.142.0.102 -n 10.142.1.98 \
--test multi-register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/multi-register/start-stop-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/start-stop-2/run_1
(test_impl.go:291).Fatal: output in run_122045.073594683_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.124 -n 10.142.1.149 -n 10.142.1.116 -n 10.142.1.125 -n 10.142.1.133 \
--test multi-register --nemesis start-stop-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/multi-register/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_122045.845345901_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.87 -n 10.142.0.109 -n 10.142.1.107 -n 10.142.1.89 -n 10.142.1.109 \
--test multi-register --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/register/majority-ring failed due to test artifacts and logs in: /artifacts/jepsen/register/majority-ring/run_1
(test_impl.go:291).Fatal: output in run_122214.185656775_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.52 -n 10.142.1.104 -n 10.142.1.151 -n 10.142.1.160 -n 10.142.1.150 \
--test register --nemesis majority-ring > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/multi-register/strobe-skews failed due to test artifacts and logs in: /artifacts/jepsen/multi-register/strobe-skews/run_1
(test_impl.go:291).Fatal: output in run_122222.681493684_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.0.188 -n 10.142.1.42 -n 10.142.0.148 -n 10.142.1.19 -n 10.142.0.177 \
--test multi-register --nemesis strobe-skews > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/register/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_122730.701410140_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.0.237 -n 10.142.0.221 -n 10.142.1.46 -n 10.142.1.62 -n 10.142.1.40 \
--test register --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/register/parts-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/parts-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_122912.286840348_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.0.222 -n 10.142.0.249 -n 10.142.1.5 -n 10.142.0.46 -n 10.142.0.253 \
--test register --nemesis parts --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/register/split failed due to test artifacts and logs in: /artifacts/jepsen/register/split/run_1
(test_impl.go:291).Fatal: output in run_123006.706704713_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.55 -n 10.142.1.119 -n 10.142.1.9 -n 10.142.1.61 -n 10.142.1.21 \
--test register --nemesis split > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/register/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_123038.685621713_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.0.62 -n 10.142.0.43 -n 10.142.0.96 -n 10.142.0.169 -n 10.142.0.51 \
--test register --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/monotonic/start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/monotonic/start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_120815.661539339_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.231 -n 10.142.1.228 -n 10.142.0.78 -n 10.142.1.229 -n 10.142.0.2 \
--test monotonic --nemesis start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/register/start-stop-2 failed due to test artifacts and logs in: /artifacts/jepsen/register/start-stop-2/run_1
(test_impl.go:291).Fatal: output in run_123126.701248194_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.71 -n 10.142.0.196 -n 10.142.0.172 -n 10.142.0.191 -n 10.142.1.128 \
--test register --nemesis start-stop-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/register/strobe-skews failed due to test artifacts and logs in: /artifacts/jepsen/register/strobe-skews/run_1
(test_impl.go:291).Fatal: output in run_123604.421276100_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.0.91 -n 10.142.0.95 -n 10.142.0.86 -n 10.142.0.89 -n 10.142.0.57 \
--test register --nemesis strobe-skews > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!
roachtest.ssh_problem failed with artifacts on master @ 8357abb668a5adaff781343b394b162fb1b66c6e:
test jepsen/sequential/majority-ring-start-kill-2 failed due to test artifacts and logs in: /artifacts/jepsen/sequential/majority-ring-start-kill-2/run_1
(test_impl.go:291).Fatal: output in run_123834.605695740_n6_bash: bash -e -c "cd /mnt/data1/jepsen/cockroachdb && set -eo pipefail && java -jar cockroachdb-0.2.0-1150b38f-standalone.jar test \
--tarball file://${PWD}/cockroach.tgz \
--username ${USER} \
--ssh-private-key ~/.ssh/id_rsa \
--os ubuntu \
--time-limit 300 \
--concurrency 30 \
--recovery-time 25 \
--test-count 1 \
-n 10.142.1.94 -n 10.142.1.59 -n 10.142.0.236 -n 10.142.0.61 -n 10.142.0.107 \
--test sequential --nemesis majority-ring --nemesis2 start-kill-2 > invoke.log 2>&1" returned: error persisted after 3 attempts: SSH_PROBLEM: exit status 255
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_encrypted=false
, ROACHTEST_fs=ext4
, ROACHTEST_localSSD=true
, ROACHTEST_ssd=0
This test on roachdash | Improve this report!