cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

roachtest: tpcc/headroom/isolation-level=read-committed/n4cpu16 failed

Open cockroach-teamcity opened this issue 9 months ago • 9 comments

roachtest.tpcc/headroom/isolation-level=read-committed/n4cpu16 failed with artifacts on master @ b185402d34768c898d63a61de3e585e4723fbe81:

(monitor.go:154).Wait: monitor failure: full command output in run_134510.010927904_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-38711

cockroach-teamcity avatar May 14 '24 13:05 cockroach-teamcity

Error: error in delivery: select order_line failed: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): "sql txn" meta={id=6cc83439 key=/Table/111/1/525/1 iso=ReadCommitted pri=0.00929437 epo=0 ts=1715694767.408104864,9 min=1715694761.690983060,18 seq=5} lock=true stat=PENDING rts=1715694767.408104864,9 wto=false gul=1715694762.190983060,18 (SQLSTATE 40001)

Reassigning to KV to take a look.

renatolabs avatar May 15 '24 07:05 renatolabs

@arulajmani Mind taking a look at this? Is this a legit transaction abort that should just be retried, or some ReadCommitted misuse?

pav-kv avatar May 15 '24 12:05 pav-kv

We run these read committed TPCC versions without retry loops as we don't expect retry errors to ever make it back to the client. It seems like we're encountering a case where this isn't true -- I'll take a look.

arulajmani avatar May 15 '24 13:05 arulajmani

We have marked this test failure issue as stale because it has been inactive for 1 month. If this failure is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the test failure queue tidy.

github-actions[bot] avatar Jun 17 '24 10:06 github-actions[bot]

roachtest.tpcc/headroom/isolation-level=read-committed/n4cpu16 failed with artifacts on master @ 67b4af76ba20ed2f6935da31eda7dfe8fc0b63e2:

(monitor.go:154).Wait: monitor failure: full command output in run_132551.700355933_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jun 19 '24 15:06 cockroach-teamcity

roachtest.tpcc/headroom/isolation-level=read-committed/n4cpu16 failed with artifacts on master @ 047a7ed79756eef53b8b9ab4c9dd9c5a463496c9:

(cluster.go:2417).Run: full command output in run_133904.532315086_n4_tpcc-workload-check-.log: COMMAND_PROBLEM: exit status 127
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jun 29 '24 13:06 cockroach-teamcity

roachtest.tpcc/headroom/isolation-level=read-committed/n4cpu16 failed with artifacts on master @ 047a7ed79756eef53b8b9ab4c9dd9c5a463496c9:

(cluster.go:2417).Run: full command output in run_151734.911289954_n4_tpcc-workload-check-.log: COMMAND_PROBLEM: exit status 127
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jun 29 '24 15:06 cockroach-teamcity

roachtest.tpcc/headroom/isolation-level=read-committed/n4cpu16 failed with artifacts on master @ d13253527955eaa2da09394b8a2729627ab25c48:

(cluster.go:2417).Run: full command output in run_130829.255490094_n4_tpcc-workload-check-.log: COMMAND_PROBLEM: exit status 127
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jun 30 '24 13:06 cockroach-teamcity

roachtest.tpcc/headroom/isolation-level=read-committed/n4cpu16 failed with artifacts on master @ d13253527955eaa2da09394b8a2729627ab25c48:

(cluster.go:2417).Run: full command output in run_153146.495406942_n4_tpcc-workload-check-.log: COMMAND_PROBLEM: exit status 127
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jun 30 '24 15:06 cockroach-teamcity

roachtest.tpcc/headroom/isolation-level=read-committed/n4cpu16 failed with artifacts on master @ d13253527955eaa2da09394b8a2729627ab25c48:

(cluster.go:2417).Run: full command output in run_141029.917821734_n4_tpcc-workload-check-.log: COMMAND_PROBLEM: exit status 127
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=true
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 01 '24 14:07 cockroach-teamcity

The 5 most recent failures (edit: and also the one after this comment) are resolved by https://github.com/cockroachdb/cockroach/pull/126453; leaving this open for the original issue.

rafiss avatar Jul 01 '24 15:07 rafiss

roachtest.tpcc/headroom/isolation-level=read-committed/n4cpu16 failed with artifacts on master @ d13253527955eaa2da09394b8a2729627ab25c48:

(cluster.go:2417).Run: full command output in run_153105.102556366_n4_tpcc-workload-check-.log: COMMAND_PROBLEM: exit status 127
test artifacts and logs in: /artifacts/tpcc/headroom/isolation-level=read-committed/n4cpu16/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 01 '24 15:07 cockroach-teamcity

We have marked this test failure issue as stale because it has been inactive for 1 month. If this failure is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the test failure queue tidy.

github-actions[bot] avatar Aug 01 '24 10:08 github-actions[bot]

I think this is the same as https://github.com/cockroachdb/cockroach/issues/127811. The artifacts are gone, but the ABORT_REASON_CLIENT_REJECT lets us make an educated guess. I'll let this close out in favour of the other issue.

arulajmani avatar Aug 01 '24 13:08 arulajmani