cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

cli: Example_demo_locality failed ["change replicas of r61 failed: descriptor changed" during SPLIT]

Open cockroach-teamcity opened this issue 3 years ago • 6 comments

cli.Example_demo_locality failed with artifacts on master @ 54b4fcaa231abd16260fb09722ed291dbd6a4900:

=== RUN   Example_demo_locality
--- FAIL: Example_demo_locality (14.66s)
got:
demo --nodes 3 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
demo --nodes 9 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
ERROR: executing ALTER TABLE vehicles SPLIT AT VALUES ('san francisco','88888888-8888-4800-8000-000000000008'): pq: change replicas of r61 failed: descriptor changed: [expected] r61:/{Table/106/1/"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f"-Max} [(n2,s2):1, (n7,s7):2, (n5,s5):3VOTER_DEMOTING_LEARNER, (n6,s6):4VOTER_INCOMING, next=5, gen=13, sticky=9223372036.854775807,2147483647] != [actual] r61:/{Table/106/1/"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f"-Max} [(n2,s2):1, (n7,s7):2, (n6,s6):4VOTER_DEMOTING_LEARNER, (n5,s5):5VOTER_INCOMING, next=6, gen=17, sticky=9223372036.854775807,2147483647]
demo --nodes 3 --multitenant=false --demo-locality=region=us-east1:region=us-east2:region=us-east3 -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1
2	region=us-east2
3	region=us-east3
want:
demo --nodes 3 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
demo --nodes 9 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
4	region=us-west1,az=a
5	region=us-west1,az=b
6	region=us-west1,az=c
7	region=europe-west1,az=b
8	region=europe-west1,az=c
9	region=europe-west1,az=d
demo --nodes 3 --multitenant=false --demo-locality=region=us-east1:region=us-east2:region=us-east3 -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1
2	region=us-east2
3	region=us-east3

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-experience @cockroachdb/server @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-17471

cockroach-teamcity avatar Jul 09 '22 05:07 cockroach-teamcity

error is:

ERROR: executing ALTER TABLE vehicles SPLIT AT VALUES ('san francisco','88888888-8888-4800-8000-000000000008'): pq: change replicas of r61 failed: descriptor changed: [expected] r61:/{Table/106/1/"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f"-Max} [(n2,s2):1, (n7,s7):2, (n5,s5):3VOTER_DEMOTING_LEARNER, (n6,s6):4VOTER_INCOMING, next=5, gen=13, sticky=9223372036.854775807,2147483647] != [actual] r61:/{Table/106/1/"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f"-Max} [(n2,s2):1, (n7,s7):2, (n6,s6):4VOTER_DEMOTING_LEARNER, (n5,s5):5VOTER_INCOMING, next=6, gen=17, sticky=9223372036.854775807,2147483647]

adityamaru avatar Jul 12 '22 14:07 adityamaru

Another failure was seen https://teamcity.cockroachdb.com/viewLog.html?buildId=5724616&buildTypeId=Cockroach_GitHubCiOptional.

adityamaru avatar Jul 12 '22 14:07 adityamaru

cli.Example_demo_locality failed with artifacts on master @ 88d3253301457ac57820e0f4a4fab8f74bf9f38b:

=== RUN   Example_demo_locality
--- FAIL: Example_demo_locality (14.81s)
got:
demo --nodes 3 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
demo --nodes 9 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
ERROR: executing ALTER TABLE users SPLIT AT VALUES ('new york','19999999-9999-4a00-8000-000000000005'): pq: change replicas of r53 failed: descriptor changed: [expected] r53:/Table/106/1/"{los angeles"/"\x99\x99\x99\x99\x99\x99H\x00\x80\x00\x00\x00\x00\x00\x00\x1e"-paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00("} [(n1,s1):1, (n4,s4):2VOTER_DEMOTING_LEARNER, (n9,s9):3, (n6,s6):4VOTER_INCOMING, next=5, gen=8, sticky=9223372036.854775807,2147483647] != [actual] r53:/Table/106/1/"{los angeles"/"\x99\x99\x99\x99\x99\x99H\x00\x80\x00\x00\x00\x00\x00\x00\x1e"-paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00("} [(n1,s1):1, (n6,s6):4VOTER_DEMOTING_LEARNER, (n9,s9):3, (n4,s4):5VOTER_INCOMING, next=6, gen=12, sticky=9223372036.854775807,2147483647]
demo --nodes 3 --multitenant=false --demo-locality=region=us-east1:region=us-east2:region=us-east3 -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1
2	region=us-east2
3	region=us-east3
want:
demo --nodes 3 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
demo --nodes 9 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
4	region=us-west1,az=a
5	region=us-west1,az=b
6	region=us-west1,az=c
7	region=europe-west1,az=b
8	region=europe-west1,az=c
9	region=europe-west1,az=d
demo --nodes 3 --multitenant=false --demo-locality=region=us-east1:region=us-east2:region=us-east3 -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1
2	region=us-east2
3	region=us-east3

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 14 '22 05:07 cockroach-teamcity

cli.Example_demo_locality failed with artifacts on master @ eabdc49383a19d5731b765dbb0d8b45bd9e24404:

=== RUN   Example_demo_locality
--- FAIL: Example_demo_locality (16.71s)
got:
demo --nodes 3 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
demo --nodes 9 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
ERROR: executing ALTER TABLE users SPLIT AT VALUES ('seattle','66666666-6666-4800-8000-000000000014'): pq: change replicas of r52 failed: descriptor changed: [expected] r52:/{Table/106/1/"paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00("-Max} [(n1,s1):1, (n9,s9):2VOTER_DEMOTING_LEARNER, (n5,s5):3, (n7,s7):4VOTER_INCOMING, next=5, gen=7, sticky=9223372036.854775807,2147483647] != [actual] r52:/{Table/106/1/"paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00("-Max} [(n2,s2):7, (n8,s8):5, (n4,s4):6VOTER_DEMOTING_LEARNER, (n6,s6):8VOTER_INCOMING, next=9, gen=23, sticky=9223372036.854775807,2147483647]
demo --nodes 3 --multitenant=false --demo-locality=region=us-east1:region=us-east2:region=us-east3 -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1
2	region=us-east2
3	region=us-east3
want:
demo --nodes 3 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
demo --nodes 9 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
4	region=us-west1,az=a
5	region=us-west1,az=b
6	region=us-west1,az=c
7	region=europe-west1,az=b
8	region=europe-west1,az=c
9	region=europe-west1,az=d
demo --nodes 3 --multitenant=false --demo-locality=region=us-east1:region=us-east2:region=us-east3 -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1
2	region=us-east2
3	region=us-east3

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Jul 25 '22 06:07 cockroach-teamcity

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_GitHubCiOptional/5909633?showRootCauses=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

This needs a retry loop around the splits or something

ajwerner avatar Jul 29 '22 16:07 ajwerner

cli.Example_demo_locality failed with artifacts on master @ d266989c4d30e6fbdc3fd6ac7458e7a651b87131:

=== RUN   Example_demo_locality
--- FAIL: Example_demo_locality (21.34s)
got:
demo --nodes 3 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
demo --nodes 9 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
ERROR: executing ALTER TABLE vehicles SPLIT AT VALUES ('new york','11111111-1111-4100-8000-000000000001'): pq: change replicas of r64 failed: descriptor changed: [expected] r64:/Table/10{6/1/"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f"-7/1/"san francisco"/"\x88\x88\x88\x88\x88\x88H\x00\x80\x00\x00\x00\x00\x00\x00\b"} [(n3,s3):1, (n4,s4):2, (n7,s7):3VOTER_DEMOTING_LEARNER, (n9,s9):4VOTER_INCOMING, next=5, gen=14, sticky=9223372036.854775807,2147483647] != [actual] r64:/Table/10{6/1/"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f"-7/1/"san francisco"/"\x88\x88\x88\x88\x88\x88H\x00\x80\x00\x00\x00\x00\x00\x00\b"} [(n3,s3):1, (n4,s4):2VOTER_DEMOTING_LEARNER, (n9,s9):4, (n5,s5):5VOTER_INCOMING, next=6, gen=18, sticky=9223372036.854775807,2147483647]
demo --nodes 3 --multitenant=false --demo-locality=region=us-east1:region=us-east2:region=us-east3 -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1
2	region=us-east2
3	region=us-east3
want:
demo --nodes 3 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
demo --nodes 9 --multitenant=false -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1,az=b
2	region=us-east1,az=c
3	region=us-east1,az=d
4	region=us-west1,az=a
5	region=us-west1,az=b
6	region=us-west1,az=c
7	region=europe-west1,az=b
8	region=europe-west1,az=c
9	region=europe-west1,az=d
demo --nodes 3 --multitenant=false --demo-locality=region=us-east1:region=us-east2:region=us-east3 -e select node_id, locality from crdb_internal.gossip_nodes order by node_id
node_id	locality
1	region=us-east1
2	region=us-east2
3	region=us-east3

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Aug 11 '22 05:08 cockroach-teamcity

We have marked this test failure issue as stale because it has been inactive for 1 month. If this failure is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the test failure queue tidy.

github-actions[bot] avatar Sep 12 '22 10:09 github-actions[bot]