elasticsearch
elasticsearch copied to clipboard
Published
20 hours ago •
elastic
elastic
Fix testStoragePreventsAllocation
startRandomShards was randomly causing shard moves (including on the later iterations) that were unexpected by test. This is fixed by disabling them on the final iterations and allowing enough time for the shard allocation to stabilize.
Closes: https://github.com/elastic/elasticsearch/issues/105957
I am not sure I follow why this failed still. I think on the second last round we should only see one shard being started and thus no relocations on that round?
This particular seed generated 3 hot nodes and 6 shards (3 primary and 3 replicas). Within round 0 we allocated all primaries evenly. Within round 1 we allocated all replicas evenly.
Round 1 and every subsequent round generated a random movement. Since cluster was balanced before that, allocator was generating a rebalance shard move to counteract every previous shard movement. This way we end up having 2 initializing shard every round (1 from forced move and 1 from counteracting balancing).
History of the shard movements in the test
[2024-05-16T05:28:14,480][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] before test
[2024-05-16T05:28:14,743][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Round: 0
[2024-05-16T05:28:14,771][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] routing_nodes:
-----node_id[hot_1][V]
--------[test-3][0], node[hot_1], [P], recovery_source[new shard recovery], s[INITIALIZING], a[id=1-pBgSlcQ7upaoAE22L5hw], unassigned_info[[reason=INDEX_CREATED], at[2024-05-16T07:28:14.553Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0], expected_shard_size[1]
-----node_id[hot_0][V]
--------[test-3][1], node[hot_0], [P], recovery_source[new shard recovery], s[INITIALIZING], a[id=n_AsUZhqS_Gag56HO7_vlw], unassigned_info[[reason=INDEX_CREATED], at[2024-05-16T07:28:14.553Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0], expected_shard_size[1]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][2], node[hot_2], [P], recovery_source[new shard recovery], s[INITIALIZING], a[id=OXCFFPUIScOPsMYk2X1B7g], unassigned_info[[reason=INDEX_CREATED], at[2024-05-16T07:28:14.553Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0], expected_shard_size[1]
---- unassigned
[2024-05-16T05:28:14,845][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Round: 1
[2024-05-16T05:28:14,850][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] routing_nodes:
-----node_id[hot_1][V]
--------[test-3][0], node[hot_1], [P], s[STARTED], a[id=1-pBgSlcQ7upaoAE22L5hw], failed_attempts[0]
--------[test-3][2], node[hot_1], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=H8SsbElpRfyWd6bSSVEfQg], unassigned_info[[reason=INDEX_CREATED], at[2024-05-16T07:28:14.553Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0], expected_shard_size[1]
-----node_id[hot_0][V]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
--------[test-3][0], node[hot_0], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=iRO79jAjTbSyNTVYzKCOCQ], unassigned_info[[reason=INDEX_CREATED], at[2024-05-16T07:28:14.553Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0], expected_shard_size[1]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=OXCFFPUIScOPsMYk2X1B7g], failed_attempts[0]
--------[test-3][1], node[hot_2], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=muXUNtu8SGmorX_E9gI8iQ], unassigned_info[[reason=INDEX_CREATED], at[2024-05-16T07:28:14.553Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0], expected_shard_size[1]
---- unassigned
[2024-05-16T05:28:14,851][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Moving [test-3][2][P] from hot_2 to hot_0
[2024-05-16T05:28:14,889][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Round: 2
[2024-05-16T05:28:14,897][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] routing_nodes:
-----node_id[hot_1][V]
--------[test-3][0], node[hot_1], [P], s[STARTED], a[id=1-pBgSlcQ7upaoAE22L5hw], failed_attempts[0]
--------[test-3][2], node[hot_1], [R], s[STARTED], a[id=H8SsbElpRfyWd6bSSVEfQg], failed_attempts[0]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=iRO79jAjTbSyNTVYzKCOCQ], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
--------[test-3][2], node[hot_0], relocating [hot_2], [P], s[RELOCATING], a[id=aWk21IFPSAO_L9SwzCAfTA, rId=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0], expected_shard_size[1]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][1], node[hot_2], [R], s[STARTED], a[id=muXUNtu8SGmorX_E9gI8iQ], failed_attempts[0]
--------[test-3][2], node[hot_2], relocating [hot_0], [P], recovery_source[peer recovery], s[INITIALIZING], a[id=XCo3ZuRZTiqIbtogiSO1ng, rId=aWk21IFPSAO_L9SwzCAfTA], failed_attempts[0], expected_shard_size[1]
---- unassigned
[2024-05-16T05:28:14,898][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Moving [test-3][1][R] from hot_2 to hot_1
[2024-05-16T05:28:14,939][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Round: 3
[2024-05-16T05:28:14,942][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] routing_nodes:
-----node_id[hot_1][V]
--------[test-3][0], node[hot_1], [P], s[STARTED], a[id=1-pBgSlcQ7upaoAE22L5hw], failed_attempts[0]
--------[test-3][1], node[hot_1], relocating [hot_2], [R], s[RELOCATING], a[id=AxpwlEYvR9KUm6AzgrMcXA, rId=x3inqX2ZQy2iV5dwEzfQ7g], failed_attempts[0], expected_shard_size[1]
--------[test-3][2], node[hot_1], [R], s[STARTED], a[id=H8SsbElpRfyWd6bSSVEfQg], failed_attempts[0]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=iRO79jAjTbSyNTVYzKCOCQ], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
--------[test-3][1], node[hot_2], relocating [hot_1], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=x3inqX2ZQy2iV5dwEzfQ7g, rId=AxpwlEYvR9KUm6AzgrMcXA], failed_attempts[0], expected_shard_size[1]
---- unassigned
[2024-05-16T05:28:14,942][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Moving [test-3][2][R] from hot_1 to hot_0
[2024-05-16T05:28:14,986][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Round: 4
[2024-05-16T05:28:14,989][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] routing_nodes:
-----node_id[hot_1][V]
--------[test-3][0], node[hot_1], [P], s[STARTED], a[id=1-pBgSlcQ7upaoAE22L5hw], failed_attempts[0]
--------[test-3][2], node[hot_1], relocating [hot_0], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=jHZjDm71TJiLSD_LYSojFQ, rId=ZcmaXp0cTrmK7rTMH_jlMQ], failed_attempts[0], expected_shard_size[1]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=iRO79jAjTbSyNTVYzKCOCQ], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
--------[test-3][2], node[hot_0], relocating [hot_1], [R], s[RELOCATING], a[id=ZcmaXp0cTrmK7rTMH_jlMQ, rId=jHZjDm71TJiLSD_LYSojFQ], failed_attempts[0], expected_shard_size[1]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][1], node[hot_2], [R], s[STARTED], a[id=x3inqX2ZQy2iV5dwEzfQ7g], failed_attempts[0]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
---- unassigned
[2024-05-16T05:28:14,989][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Moving [test-3][0][P] from hot_1 to hot_2
[2024-05-16T05:28:15,024][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Round: 5
[2024-05-16T05:28:15,027][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] routing_nodes:
-----node_id[hot_1][V]
--------[test-3][2], node[hot_1], [R], s[STARTED], a[id=jHZjDm71TJiLSD_LYSojFQ], failed_attempts[0]
--------[test-3][1], node[hot_1], relocating [hot_2], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=5R8inIH2RNKaZdTae2Yh1Q, rId=x3inqX2ZQy2iV5dwEzfQ7g], failed_attempts[0], expected_shard_size[1]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=iRO79jAjTbSyNTVYzKCOCQ], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=d2Pr35-sTR6xaDOFKvPAxg], failed_attempts[0]
--------[test-3][1], node[hot_2], relocating [hot_1], [R], s[RELOCATING], a[id=x3inqX2ZQy2iV5dwEzfQ7g, rId=5R8inIH2RNKaZdTae2Yh1Q], failed_attempts[0], expected_shard_size[1]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
---- unassigned
[2024-05-16T05:28:15,028][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Moving [test-3][0][R] from hot_0 to hot_1
[2024-05-16T05:28:15,065][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Round: 6
[2024-05-16T05:28:15,068][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] routing_nodes:
-----node_id[hot_1][V]
--------[test-3][0], node[hot_1], [R], s[STARTED], a[id=y3ccqxwZTW-EteHIhY7V9w], failed_attempts[0]
--------[test-3][1], node[hot_1], [R], s[STARTED], a[id=5R8inIH2RNKaZdTae2Yh1Q], failed_attempts[0]
--------[test-3][2], node[hot_1], relocating [hot_0], [R], s[RELOCATING], a[id=jHZjDm71TJiLSD_LYSojFQ, rId=I9kP9ozzQBm3rpuBcPSXHg], failed_attempts[0], expected_shard_size[1]
-----node_id[hot_0][V]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
--------[test-3][2], node[hot_0], relocating [hot_1], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=I9kP9ozzQBm3rpuBcPSXHg, rId=jHZjDm71TJiLSD_LYSojFQ], failed_attempts[0], expected_shard_size[1]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=d2Pr35-sTR6xaDOFKvPAxg], failed_attempts[0]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
---- unassigned
[2024-05-16T05:28:15,068][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Moving [test-3][0][R] from hot_1 to hot_0
[2024-05-16T05:28:15,101][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Round: 7
[2024-05-16T05:28:15,104][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] routing_nodes:
-----node_id[hot_1][V]
--------[test-3][1], node[hot_1], [R], s[STARTED], a[id=5R8inIH2RNKaZdTae2Yh1Q], failed_attempts[0]
--------[test-3][2], node[hot_1], relocating [hot_0], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=arwS8drfTUGwIqtroigDSg, rId=I9kP9ozzQBm3rpuBcPSXHg], failed_attempts[0], expected_shard_size[1]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=atyBChlRQHO71geXDU8vWA], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
--------[test-3][2], node[hot_0], relocating [hot_1], [R], s[RELOCATING], a[id=I9kP9ozzQBm3rpuBcPSXHg, rId=arwS8drfTUGwIqtroigDSg], failed_attempts[0], expected_shard_size[1]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=d2Pr35-sTR6xaDOFKvPAxg], failed_attempts[0]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
---- unassigned
[2024-05-16T05:28:15,104][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Moving [test-3][1][R] from hot_1 to hot_2
[2024-05-16T05:28:15,128][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] Round: 8
[2024-05-16T05:28:15,131][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] routing_nodes:
-----node_id[hot_1][V]
--------[test-3][2], node[hot_1], [R], s[STARTED], a[id=arwS8drfTUGwIqtroigDSg], failed_attempts[0]
--------[test-3][1], node[hot_1], relocating [hot_2], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=qVJFf9gGQMahNC2gL_TNQA, rId=pxYIe0NsTUKiynUkxsXKyg], failed_attempts[0], expected_shard_size[1]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=atyBChlRQHO71geXDU8vWA], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=d2Pr35-sTR6xaDOFKvPAxg], failed_attempts[0]
--------[test-3][1], node[hot_2], relocating [hot_1], [R], s[RELOCATING], a[id=pxYIe0NsTUKiynUkxsXKyg, rId=qVJFf9gGQMahNC2gL_TNQA], failed_attempts[0], expected_shard_size[1]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
---- unassigned
[2024-05-16T05:28:15,139][INFO ][o.e.x.a.s.ReactiveStorageDeciderDecisionTests] [testStoragePreventsAllocation] after test
REPRODUCE WITH: ./gradlew 'null' --tests "org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageDeciderDecisionTests.testStoragePreventsAllocation" -Dtests.seed=2BCE202E19A4B5D1 -Dtests.locale=sr-ME -Dtests.timezone=Etc/GMT+2 -Druntime.java=17
NOTE: leaving temporary files on disk at: /tmp/org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageDeciderDecisionTests_2BCE202E19A4B5D1-005
java.lang.AssertionError:
Expected: sameInstance(<cluster uuid: _na_ [committed: false]
version: 0
state uuid: _iIQhRdeRty5glA_ErGuGQ
from_diff: false
meta data version: 0
coordination_metadata:
term: 0
last_committed_config: VotingConfiguration{}
last_accepted_config: VotingConfiguration{}
voting tombstones: []
[test-3]: v[9], mv[1], sv[1], av[1]
0: p_term [1], isa_ids [d2Pr35-sTR6xaDOFKvPAxg, atyBChlRQHO71geXDU8vWA]
1: p_term [1], isa_ids [n_AsUZhqS_Gag56HO7_vlw, 5R8inIH2RNKaZdTae2Yh1Q]
2: p_term [1], isa_ids [XCo3ZuRZTiqIbtogiSO1ng, I9kP9ozzQBm3rpuBcPSXHg]
metadata customs:
index-graveyard: IndexGraveyard[[]]
nodes (node-left generation: 0):
{warm_0}{warm_0}{1EYZQvruRniPiEW8YHC5iw}{MjI4EVhjTe-lsxlN4cc1aA}{0.0.0.0}{0.0.0.0:4}{w}{8.15.0}{7000099-8506000}
{warm_1}{warm_1}{Mk_JeZlWT5OBCG4LhRKO6A}{IwFd-H3HRk2QduaMocAh1w}{0.0.0.0}{0.0.0.0:5}{w}{8.15.0}{7000099-8506000}
{warm_2}{warm_2}{6qUxxIEAQvmKNv7qXDzpIw}{jt6yMqEITtGszlN6WjNFEQ}{0.0.0.0}{0.0.0.0:6}{w}{8.15.0}{7000099-8506000}
{hot_0}{hot_0}{Yfk1iYruQKuu_0BkpvbqcA}{qflksi39TUyqDwGrPymUEA}{0.0.0.0}{0.0.0.0:1}{h}{8.15.0}{7000099-8506000}
{hot_1}{hot_1}{JoLREcpbQtOmEKaYmQR5Dw}{nZGiuvWaTsOawg_jDkpM_g}{0.0.0.0}{0.0.0.0:2}{h}{8.15.0}{7000099-8506000}
{hot_2}{hot_2}{S888U6qpQ92GLNOYfI5ksw}{mdFdHJLOTJutb7aacDST2g}{0.0.0.0}{0.0.0.0:3}{h}{8.15.0}{7000099-8506000}
cluster features:
warm_0: []
warm_1: []
warm_2: []
hot_0: []
hot_1: []
hot_2: []
routing_table (version 0):
-- index [[test-3]]
----shard_id [test-3][0]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=atyBChlRQHO71geXDU8vWA], failed_attempts[0]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=d2Pr35-sTR6xaDOFKvPAxg], failed_attempts[0]
----shard_id [test-3][1]
--------[test-3][1], node[hot_1], relocating [hot_2], [R], s[RELOCATING], a[id=5R8inIH2RNKaZdTae2Yh1Q, rId=pxYIe0NsTUKiynUkxsXKyg], failed_attempts[0], expected_shard_size[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
----shard_id [test-3][2]
--------[test-3][2], node[hot_0], relocating [hot_1], [R], s[RELOCATING], a[id=I9kP9ozzQBm3rpuBcPSXHg, rId=arwS8drfTUGwIqtroigDSg], failed_attempts[0], expected_shard_size[1]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
routing_nodes:
-----node_id[hot_1][V]
--------[test-3][1], node[hot_1], relocating [hot_2], [R], s[RELOCATING], a[id=5R8inIH2RNKaZdTae2Yh1Q, rId=pxYIe0NsTUKiynUkxsXKyg], failed_attempts[0], expected_shard_size[0]
--------[test-3][2], node[hot_1], relocating [hot_0], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=arwS8drfTUGwIqtroigDSg, rId=I9kP9ozzQBm3rpuBcPSXHg], failed_attempts[0], expected_shard_size[1]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=atyBChlRQHO71geXDU8vWA], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
--------[test-3][2], node[hot_0], relocating [hot_1], [R], s[RELOCATING], a[id=I9kP9ozzQBm3rpuBcPSXHg, rId=arwS8drfTUGwIqtroigDSg], failed_attempts[0], expected_shard_size[1]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=d2Pr35-sTR6xaDOFKvPAxg], failed_attempts[0]
--------[test-3][1], node[hot_2], relocating [hot_1], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=pxYIe0NsTUKiynUkxsXKyg, rId=5R8inIH2RNKaZdTae2Yh1Q], failed_attempts[0], expected_shard_size[0]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
---- unassigned
>)
but: was <cluster uuid: _na_ [committed: false]
version: 0
state uuid: _iIQhRdeRty5glA_ErGuGQ
from_diff: false
meta data version: 0
coordination_metadata:
term: 0
last_committed_config: VotingConfiguration{}
last_accepted_config: VotingConfiguration{}
voting tombstones: []
[test-3]: v[10], mv[1], sv[1], av[1]
0: p_term [1], isa_ids [d2Pr35-sTR6xaDOFKvPAxg, atyBChlRQHO71geXDU8vWA]
1: p_term [1], isa_ids [n_AsUZhqS_Gag56HO7_vlw, pxYIe0NsTUKiynUkxsXKyg]
2: p_term [1], isa_ids [XCo3ZuRZTiqIbtogiSO1ng, arwS8drfTUGwIqtroigDSg]
metadata customs:
index-graveyard: IndexGraveyard[[]]
nodes (node-left generation: 0):
{warm_0}{warm_0}{1EYZQvruRniPiEW8YHC5iw}{MjI4EVhjTe-lsxlN4cc1aA}{0.0.0.0}{0.0.0.0:4}{w}{8.15.0}{7000099-8506000}
{warm_1}{warm_1}{Mk_JeZlWT5OBCG4LhRKO6A}{IwFd-H3HRk2QduaMocAh1w}{0.0.0.0}{0.0.0.0:5}{w}{8.15.0}{7000099-8506000}
{warm_2}{warm_2}{6qUxxIEAQvmKNv7qXDzpIw}{jt6yMqEITtGszlN6WjNFEQ}{0.0.0.0}{0.0.0.0:6}{w}{8.15.0}{7000099-8506000}
{hot_0}{hot_0}{Yfk1iYruQKuu_0BkpvbqcA}{qflksi39TUyqDwGrPymUEA}{0.0.0.0}{0.0.0.0:1}{h}{8.15.0}{7000099-8506000}
{hot_1}{hot_1}{JoLREcpbQtOmEKaYmQR5Dw}{nZGiuvWaTsOawg_jDkpM_g}{0.0.0.0}{0.0.0.0:2}{h}{8.15.0}{7000099-8506000}
{hot_2}{hot_2}{S888U6qpQ92GLNOYfI5ksw}{mdFdHJLOTJutb7aacDST2g}{0.0.0.0}{0.0.0.0:3}{h}{8.15.0}{7000099-8506000}
cluster features:
warm_0: []
warm_1: []
warm_2: []
hot_0: []
hot_1: []
hot_2: []
routing_table (version 0):
-- index [[test-3]]
----shard_id [test-3][0]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=atyBChlRQHO71geXDU8vWA], failed_attempts[0]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=d2Pr35-sTR6xaDOFKvPAxg], failed_attempts[0]
----shard_id [test-3][1]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
--------[test-3][1], node[hot_2], relocating [hot_1], [R], s[RELOCATING], a[id=pxYIe0NsTUKiynUkxsXKyg, rId=qVJFf9gGQMahNC2gL_TNQA], failed_attempts[0], expected_shard_size[1]
----shard_id [test-3][2]
--------[test-3][2], node[hot_1], [R], s[STARTED], a[id=arwS8drfTUGwIqtroigDSg], failed_attempts[0]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
routing_nodes:
-----node_id[hot_1][V]
--------[test-3][1], node[hot_1], relocating [hot_2], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=qVJFf9gGQMahNC2gL_TNQA, rId=pxYIe0NsTUKiynUkxsXKyg], failed_attempts[0], expected_shard_size[1]
--------[test-3][2], node[hot_1], [R], s[STARTED], a[id=arwS8drfTUGwIqtroigDSg], failed_attempts[0]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=atyBChlRQHO71geXDU8vWA], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=n_AsUZhqS_Gag56HO7_vlw], failed_attempts[0]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=d2Pr35-sTR6xaDOFKvPAxg], failed_attempts[0]
--------[test-3][1], node[hot_2], relocating [hot_1], [R], s[RELOCATING], a[id=pxYIe0NsTUKiynUkxsXKyg, rId=qVJFf9gGQMahNC2gL_TNQA], failed_attempts[0], expected_shard_size[1]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=XCo3ZuRZTiqIbtogiSO1ng], failed_attempts[0]
---- unassigned
>
<Click to see difference>
at __randomizedtesting.SeedInfo.seed([2BCE202E19A4B5D1:1386B774A1EB3848]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2188)
at org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageDeciderDecisionTests.testStoragePreventsAllocation(ReactiveStorageDeciderDecisionTests.java:236)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
at java.base/java.lang.Thread.run(Thread.java:833)