redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

Published 20 hours ago • verified publisher icon redpanda-data

CI Failure (`leader count on shard (2, 0) (3) is < 4`) in `AutomaticLeadershipBalancingTest.test_automatic_rebalance`

Open vbotbuildovich opened this issue 11 months ago • 27 comments

https://buildkite.com/redpanda/vtools/builds/12307

Module: rptest.tests.leadership_transfer_test
Class: AutomaticLeadershipBalancingTest
Method: test_automatic_rebalance

test_id:    AutomaticLeadershipBalancingTest.test_automatic_rebalance
status:     FAIL
run time:   91.740 seconds

AssertionError('leader count on shard (2, 0) (3) is < 4')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 104, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/leadership_transfer_test.py", line 324, in test_automatic_rebalance
    assert count >= expected_min, \
AssertionError: leader count on shard (2, 0) (3) is < 4

JIRA Link: CORE-1883

vbotbuildovich avatar Mar 16 '24 21:03 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12321

vbotbuildovich avatar Mar 17 '24 21:03 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12578 *https://buildkite.com/redpanda/redpanda/builds/47080

vbotbuildovich avatar Mar 29 '24 21:03 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12592

vbotbuildovich avatar Mar 30 '24 21:03 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/47264

vbotbuildovich avatar Apr 03 '24 07:04 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/47264

vbotbuildovich avatar Apr 03 '24 07:04 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/47372 *https://buildkite.com/redpanda/vtools/builds/12679

vbotbuildovich avatar Apr 04 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/47416 *https://buildkite.com/redpanda/redpanda/builds/47612

vbotbuildovich avatar Apr 10 '24 17:04 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/47756

vbotbuildovich avatar Apr 13 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12865 *https://buildkite.com/redpanda/vtools/builds/12866

vbotbuildovich avatar Apr 14 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/47786

vbotbuildovich avatar Apr 15 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/47838 *https://buildkite.com/redpanda/redpanda/builds/47843 *https://buildkite.com/redpanda/vtools/builds/12890

vbotbuildovich avatar Apr 16 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12994 *https://buildkite.com/redpanda/redpanda/builds/48064 *https://buildkite.com/redpanda/redpanda/builds/48086 *https://buildkite.com/redpanda/redpanda/builds/48099

vbotbuildovich avatar Apr 22 '24 22:04 vbotbuildovich

The test is failing because a newly restarted node is sending an incomplete health report (some partitions haven't started yet). I guess we can give newly restarted nodes a grace period and mute them for a bit before transferring leadership there (a good idea for other reasons as well).

ztlpn avatar Apr 23 '24 13:04 ztlpn

*https://buildkite.com/redpanda/vtools/builds/13100

vbotbuildovich avatar Apr 23 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13147

vbotbuildovich avatar Apr 24 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13189

vbotbuildovich avatar Apr 25 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13219 *https://buildkite.com/redpanda/vtools/builds/13228

vbotbuildovich avatar Apr 26 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/48366

vbotbuildovich avatar Apr 27 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/48606

vbotbuildovich avatar May 02 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/48666 *https://buildkite.com/redpanda/redpanda/builds/48667

vbotbuildovich avatar May 03 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/48717

vbotbuildovich avatar May 04 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13497

vbotbuildovich avatar May 05 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13656

vbotbuildovich avatar May 12 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/48992 *https://buildkite.com/redpanda/redpanda/builds/48990 *https://buildkite.com/redpanda/redpanda/builds/49002

vbotbuildovich avatar May 13 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/49075

vbotbuildovich avatar May 14 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/49143 *https://buildkite.com/redpanda/vtools/builds/13742

vbotbuildovich avatar May 15 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/49202 *https://buildkite.com/redpanda/vtools/builds/13769 *https://buildkite.com/redpanda/vtools/builds/13773 *https://buildkite.com/redpanda/vtools/builds/13892

vbotbuildovich avatar May 21 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/49468

vbotbuildovich avatar May 23 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/49567

vbotbuildovich avatar May 26 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/14105

vbotbuildovich avatar May 28 '24 21:05 vbotbuildovich