cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard
verified publisher icon cockroachdb

roachtest: schemachange/random-load failed [duplicate key value violates unique constraint]

Open cockroach-teamcity opened this issue 1 year ago • 4 comments

roachtest.schemachange/random-load failed with artifacts on master @ 58c475d67e32b75284b4fe293bff82807c3d129d:

(schemachange_random_load.go:123).runSchemaChangeRandomLoad: full command output in run_115201.481780467_n1_workload-run-schemac.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/random-load/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #131502 roachtest: schemachange/random-load failed [return type mismatch in function declared to return table_w16_201] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.1]
  • #131405 roachtest: schemachange/random-load failed [DROP VIEW is stuck in reverting state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2]
/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-42888

cockroach-teamcity avatar Oct 09 '24 12:10 cockroach-teamcity 

{
 "workerId": 3,
 "clientTimestamp": "11:54:23.073264",
 "ops": [
  "BEGIN",
  {
   "sql": "INSERT INTO public.table_w16_36 (col3_w16_4, col3_w16_6, col3_w16_7, col3_w16_8, col3_w16_9, col3_w16_10, col3_w16_11, col3_w16_12, col3_w16_13, col3_w16_14, col3_w16_15, col3_w16_16, col3_w16_17, col3_w16_18) VALUES ('05:03:10.72559-10:37':::TIMETZ,e'2\\x1erc~BtD'::NAME:::NAME,B'0000110001101010110011011001101011','\\x':::BYTES,(-6.524642630955600259E+29):::DECIMAL,'K:'::NAME:::NAME,'17:20:11.766973':::TIME,e'\\x02'::BPCHAR,'[null, true]':::JSONB,'93C25D81/A80E73FA':::PG_LSN,'2024-09-23 15:30:17.000506+00':::TIMESTAMPTZ,'''B'':125C,869A ''MR'':521C,741C,806A ''OqgjAss'' ''WwRmlAeY'':325A,618A,641A ''hQ'':530C ''oZpEsmM'' ''uT'' ''vWUrBTkDP'':348C,541A,679B ''zWkro''':::TSVECTOR,(-57):::INT8,e'\\x02G;Q'::NAME:::NAME),('20:11:46.832186+06:54':::TIMETZ,''::NAME:::NAME,B'1100010111010110000011100000000111','\\x1cd133e37c2b':::BYTES,71271180176854.16446:::DECIMAL,e'\\r\u003e\\t'::NAME:::NAME,'11:00:15.120591':::TIME,'V'::BPCHAR,'{\"c\": {\"a\": 4.37452864244896}, \"cAX0\": false, \"xMTlj1kg\": {\"5QCIgg\": 0.5068169918011065, \"MiVPIq\": false, \"Tw\": [true, null], \"hkmTbADmT\": true}}':::JSONB,'4F1C9A4D/A34F1B1':::PG_LSN,'2017-02-27 11:03:34.000828+00':::TIMESTAMPTZ,'''kVZFR''':::TSVECTOR,54:::INT8,''::NAME:::NAME),('19:27:32.99176+12:31':::TIMETZ,e'R@D7\\x0fJ\\x01K'::NAME:::NAME,NULL,'\\x2c':::BYTES,0:::DECIMAL,''::NAME:::NAME,'23:20:25.782725':::TIME,e'\\''::BPCHAR,'[\"Fqun\", 0.8533980135034013]':::JSONB,'8D7BD5A6/1F119A4F':::PG_LSN,'2011-06-25 08:29:47.000093+00':::TIMESTAMPTZ,'''L'':359B,779A':::TSVECTOR,84:::INT8,e'b4]\\x06'::NAME:::NAME)",
   "potentialExecErr": "23502,23514"
  }
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": "***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: duplicate key value violates unique constraint \"table_w16_3_expr_key\" (SQLSTATE 23505)",
 "errorState": {
  "potentialErrors": [
   "23502",
   "23514"
  ],

annrpom avatar Oct 09 '24 17:10 annrpom

Dang the query I was hoping to find in the list of prev statements is not there:

https://github.com/cockroachdb/cockroach/blob/15db309a564809b9e1de1b832ee13b2e0c80f237/pkg/workload/schemachange/error_screening.go#L1618-L1628

I'll try seeing if this will repro next

annrpom avatar Oct 10 '24 17:10 annrpom

https://github.com/cockroachdb/cockroach/pull/130361 was supposed to help with UNIQUE CONSTRAINT issues, but it seems like it's still flaky. https://github.com/cockroachdb/cockroach/pull/131797 was also merged last week, which addressed other workload issues, so it might be worth seeing if any are related.

rafiss avatar Oct 15 '24 17:10 rafiss

roachtest.schemachange/random-load failed with artifacts on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

(schemachange_random_load.go:123).runSchemaChangeRandomLoad: full command output in run_113938.030654397_n1_workload-run-schemac.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/random-load/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #132551 roachtest: schemachange/random-load failed [many GC for DROP SEQUENCE jobs are waiting for MVCC GC] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2.13-rc]
  • #131502 roachtest: schemachange/random-load failed [return type mismatch in function declared to return table_w16_201] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.1]
  • #131405 roachtest: schemachange/random-load failed [DROP VIEW is stuck in reverting state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 20 '24 11:10 cockroach-teamcity

roachtest.schemachange/random-load failed with artifacts on master @ 1e5b3c212b45419c960038718c48a5dd75a111a0:

(schemachange_random_load.go:123).runSchemaChangeRandomLoad: full command output in run_124304.825360361_n1_workload-run-schemac.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/random-load/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #132551 roachtest: schemachange/random-load failed [many GC for DROP SEQUENCE jobs are waiting for MVCC GC] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2.13-rc]
  • #131502 roachtest: schemachange/random-load failed [return type mismatch in function declared to return table_w16_201] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.1]
  • #131405 roachtest: schemachange/random-load failed [DROP VIEW is stuck in reverting state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 22 '24 12:10 cockroach-teamcity

roachtest.schemachange/random-load failed with artifacts on master @ 9c1d89e3adb8c6532459cc3e616288db06f966d9:

(schemachange_random_load.go:123).runSchemaChangeRandomLoad: full command output in run_122259.633992255_n1_workload-run-schemac.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/random-load/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #132551 roachtest: schemachange/random-load failed [many GC for DROP SEQUENCE jobs are waiting for MVCC GC] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2.13-rc]
  • #131502 roachtest: schemachange/random-load failed [return type mismatch in function declared to return table_w16_201] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.1]
  • #131405 roachtest: schemachange/random-load failed [DROP VIEW is stuck in reverting state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 24 '24 12:10 cockroach-teamcity

roachtest.schemachange/random-load failed with artifacts on master @ 27c521de897105cdeeed88c3a853380c14345a22:

(schemachange_random_load.go:123).runSchemaChangeRandomLoad: full command output in run_120334.358815053_n1_workload-run-schemac.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/random-load/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

  • #131405 roachtest: schemachange/random-load failed [DROP VIEW is stuck in reverting state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Nov 04 '24 12:11 cockroach-teamcity

I haven't been able to repro the duplicate key value violating unique constraint error; we have also not seen this failure in the workload since this issue

I am going to close this as in-actionable, but I would be more than happy to track this down in the workload if it occurs again

annrpom avatar Nov 14 '24 17:11 annrpom