cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

roachtest: backup/mixed-version-basic failed

Open cockroach-teamcity opened this issue 3 years ago • 18 comments

roachtest.backup/mixed-version-basic failed with artifacts on master @ 7dd1c51f6b5802e32bafd82e46747f349836592f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	backup.go:231,retry.go:207,backup.go:210,backup.go:369,versionupgrade.go:206,backup.go:572,test_runner.go:884: job failed: A full backup already exists in nodelocal://1/plan-old-resume-new/2022/05/21-052428.49. Consider running an incremental backup to this full backup via ``BACKUP INTO '/2022/05/21-052428.49' IN 'nodelocal://1/plan-old-resume-new'``
Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

Jira issue: CRDB-15971

Epic CRDB-13428

cockroach-teamcity avatar May 21 '22 05:05 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ ba1b0d2cd8557620505d5a330bb682ce82be6c2a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	backup.go:231,retry.go:207,backup.go:210,backup.go:369,versionupgrade.go:206,backup.go:572,test_runner.go:884: job failed: A full backup already exists in nodelocal://1/plan-old-resume-new/2022/05/22-052408.33. Consider running an incremental backup to this full backup via ``BACKUP INTO '/2022/05/22-052408.33' IN 'nodelocal://1/plan-old-resume-new'``
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar May 22 '22 05:05 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ 5c400e31f33b97a7b3232bb54abdd55645f0e978:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	backup.go:231,retry.go:207,backup.go:210,backup.go:369,versionupgrade.go:206,backup.go:572,test_runner.go:884: job failed: A full backup already exists in nodelocal://1/plan-old-resume-new/2022/05/23-052349.20. Consider running an incremental backup to this full backup via ``BACKUP INTO '/2022/05/23-052349.20' IN 'nodelocal://1/plan-old-resume-new'``
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar May 23 '22 05:05 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ 175eebd03e5f2674a2d1f6bfea8bd72f131a6532:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	versionupgrade.go:450,versionupgrade.go:178,backup.go:578,test_runner.go:884: 1: expected version 22.1-12, got 22.1-10

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jun 08 '22 05:06 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ 7f8fef4c3f24ec2b6f9a3db1eb38756da68b3d5a:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	versionupgrade.go:450,versionupgrade.go:178,backup.go:578,test_runner.go:896: 1: expected version 22.1-24, got 22.1-10

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jun 28 '22 05:06 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ 4e8134033890c956d39020f28a0e99b21b32649f:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	versionupgrade.go:450,versionupgrade.go:178,backup.go:578,test_runner.go:896: 1: expected version 22.1-26, got 22.1-10

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 02 '22 05:07 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ 7a8cac9b11036dcf401453eb36e531fe381b70b4:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	versionupgrade.go:450,versionupgrade.go:178,backup.go:578,test_runner.go:896: 1: expected version 22.1-26, got 22.1-10

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 05 '22 05:07 cockroach-teamcity

The failure mode on this roachtest has changed to expected version 22.1-26, got 22.1-10. This is not related to the forward port of https://github.com/cockroachdb/cockroach/issues/83660 that is pending. This should be investigated as a fresh issue. cc: @msbutler I'll try to take a look this week but pinging as current L2.

adityamaru avatar Jul 11 '22 17:07 adityamaru

roachtest.backup/mixed-version-basic failed with artifacts on master @ 571bfa3afb3858ae84d8a8fcdbb0a38e058402a5:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	assertions.go:262,assertions.go:1332,require.go:1231,backup.go:419,versionupgrade.go:178,backup.go:578,test_runner.go:896: 
			Error Trace:	backup.go:419
			            				versionupgrade.go:178
			            				backup.go:578
			            				test_runner.go:896
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.229.63.122:26257: connect: connection refused
			Test:       	backup/mixed-version-basic

	test_impl.go:265,require.go:1234,backup.go:419,versionupgrade.go:178,backup.go:578,test_runner.go:896:

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 12 '22 05:07 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ 687171ac6c2cd9992486bb3b8c9d252ac95ca1cd:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	assertions.go:262,assertions.go:1332,require.go:1231,backup.go:419,versionupgrade.go:178,backup.go:578,test_runner.go:896: 
			Error Trace:	backup.go:419
			            				versionupgrade.go:178
			            				backup.go:578
			            				test_runner.go:896
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	dial tcp 35.231.14.75:26257: connect: connection refused
			Test:       	backup/mixed-version-basic

	test_impl.go:265,require.go:1234,backup.go:419,versionupgrade.go:178,backup.go:578,test_runner.go:896:

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 13 '22 05:07 cockroach-teamcity

This latest set of test failures is caused by https://github.com/cockroachdb/cockroach/issues/84396

msbutler avatar Jul 13 '22 23:07 msbutler

roachtest.backup/mixed-version-basic failed with artifacts on master @ 88d3253301457ac57820e0f4a4fab8f74bf9f38b:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	assertions.go:262,assertions.go:1332,require.go:1231,backup.go:419,versionupgrade.go:178,backup.go:578,test_runner.go:896: 
			Error Trace:	backup.go:419
			            				versionupgrade.go:178
			            				backup.go:578
			            				test_runner.go:896
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	pq: importing 37 ranges: failed to create blob client: connecting to node 1: unable to dial n1: breaker open
			Test:       	backup/mixed-version-basic

	test_impl.go:265,require.go:1234,backup.go:419,versionupgrade.go:178,backup.go:578,test_runner.go:896:

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 14 '22 05:07 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ e9ee21860458d997a8155734dc608cfcd050ef24:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	assertions.go:262,assertions.go:1332,require.go:1231,backup.go:419,versionupgrade.go:178,backup.go:578,test_runner.go:896: 
			Error Trace:	backup.go:419
			            				versionupgrade.go:178
			            				backup.go:578
			            				test_runner.go:896
			            				asm_amd64.s:1581
			Error:      	Received unexpected error:
			            	pq: pausing due to error; use RESUME JOB to try to proceed once the issue is resolved, or CANCEL JOB to rollback: exhausted retries: importing 14 ranges: EOF
			Test:       	backup/mixed-version-basic

	test_impl.go:265,require.go:1234,backup.go:419,versionupgrade.go:178,backup.go:578,test_runner.go:896:

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 15 '22 05:07 cockroach-teamcity

Same error pebbleIterator error as https://github.com/cockroachdb/cockroach/issues/84240#issuecomment-1185820180

msbutler avatar Jul 15 '22 21:07 msbutler

Before the pebble iterator corruption bug errors, @adityamaru pointed out a different failure mode:

The failure mode on this roachtest has changed to expected version 22.1-26, got 22.1-10.

I believe the root cause of this failure relates to https://github.com/cockroachdb/cockroach/issues/84382. If this roachtest keeps flaking, we should implement the short term solution described in that issue: lower the sql descriptor lease duration. Hopefully though, schema will put out a permanent fix.

msbutler avatar Jul 20 '22 16:07 msbutler

removing release blocker, as pebble iterator corruption bug was fixed in https://github.com/cockroachdb/cockroach/pull/84666

msbutler avatar Jul 21 '22 14:07 msbutler

roachtest.backup/mixed-version-basic failed with artifacts on master @ eabdc49383a19d5731b765dbb0d8b45bd9e24404:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	versionupgrade.go:484,versionupgrade.go:188,backup.go:582,test_runner.go:896: 1: expected version 22.1-30, got 22.1-12

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 24 '22 05:07 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ ce8bfd4811e789bdd4cf31640722517a5cf46237:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	versionupgrade.go:484,versionupgrade.go:188,backup.go:582,test_runner.go:896: 1: expected version 22.1-40, got 22.1-12

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Aug 08 '22 05:08 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ 43bfc3185cb362b14e0b8a331148f0b14542a4f2:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	versionupgrade.go:484,versionupgrade.go:188,backup.go:582,test_runner.go:896: 1: expected version 22.1-44, got 22.1-12

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Aug 11 '22 05:08 cockroach-teamcity

roachtest.backup/mixed-version-basic failed with artifacts on master @ 41db784cb97d2749b162020c2c821979094f87b1:

test artifacts and logs in: /artifacts/backup/mixed-version-basic/run_1
	versionupgrade.go:484,versionupgrade.go:188,backup.go:582,test_runner.go:896: 1: expected version 22.1-58, got 22.1-12

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Aug 15 '22 05:08 cockroach-teamcity

This is still consistently failing and maybe we should do what Michael suggested above and implement https://github.com/cockroachdb/cockroach/issues/84382 so this is not hiding more sinister bugs.

adityamaru avatar Aug 26 '22 15:08 adityamaru

I ran 20 iterations of this without failure, also it seems to have passed for the past ~15 days. For posterity I ran roachtest run --count 20 backup/mixed-version-basic --cockroach=artifacts/cockroach.

Closing for now, to allow it to reopen if theres a new failure.

adityamaru avatar Aug 30 '22 19:08 adityamaru