Timeout failure on ARM PartitionBalancerTest.test_decommission

[NyaliaLui]

Version & Environment

Redpanda version: dev CDT nightly ARM:

https://buildkite.com/redpanda/vtools/builds/4201#018463f2-bed8-437b-af13-fa9cc677ecaf/6-7366 https://buildkite.com/redpanda/vtools/builds/4201#018463f2-bed8-437b-af13-fa9cc677ecaf/6-7344 https://buildkite.com/redpanda/vtools/builds/4196#01846250-d53f-473f-83be-eb6536cfa36f/6-7338 https://buildkite.com/redpanda/vtools/builds/4196#01846250-d53f-473f-83be-eb6536cfa36f/6-7316

    TimeoutError('')
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/partition_balancer_test.py", line 770, in test_decommission
    wait_until(node_removed, timeout_sec=120, backoff_sec=2)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

[andijcr]

another one https://buildkite.com/redpanda/vtools/builds/4255#018477c1-dbc9-454c-bd32-fa3b114f7df8

[ZeDRoman]

Partition reallocation is stuck here. To understand why we need trace log. I will try to reproduce it with enabled trace log

[bharathv]

again https://ci-artifacts.dev.vectorized.cloud/vtools/018526d6-f604-42f0-b676-a71840fdf989/vbuild/ducktape/results/2022-12-18--001/report.html

Multiple variations of the same test, clubbing them here rather than creating separate issues.

FAIL test: PartitionBalancerTest.test_rack_awareness (1/25 runs)
  failure at 2022-12-19T05:20:39.021Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4730#018526d6-f604-42f0-b676-a71840fdf989

FAIL test: PartitionBalancerTest.test_maintenance_mode.kill_same_node=False (1/25 runs)
  failure at 2022-12-19T05:20:39.021Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4730#018526d6-f604-42f0-b676-a71840fdf989


FAIL test: PartitionBalancerTest.test_fuzz_admin_ops (1/24 runs)
  failure at 2022-12-19T05:20:39.021Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4730#018526d6-f604-42f0-b676-a71840fdf989
FAIL test: PartitionBalancerTest.test_movement_cancellations (1/24 runs)
  failure at 2022-12-19T05:20:39.021Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4730#018526d6-f604-42f0-b676-a71840fdf989


FAIL test: PartitionBalancerTest.test_unavailable_nodes (1/25 runs)
  failure at 2022-12-19T05:20:39.021Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4730#018526d6-f604-42f0-b676-a71840fdf989


FAIL test: PartitionBalancerTest.test_rack_constraint_repair (1/25 runs)
  failure at 2022-12-19T05:20:39.021Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4730#018526d6-f604-42f0-b676-a71840fdf989

[bharathv]

https://buildkite.com/redpanda/vtools/builds/4788#01853123-39d8-452b-b909-9c057db48f19

FAIL test: PartitionBalancerTest.test_decommission.kill_same_node=False.decommission_first=True (1/36 runs)
  failure at 2022-12-21T05:00:23.156Z: TimeoutError('')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4788#01853123-39d8-452b-b909-9c057db48f19

FAIL test: PartitionBalancerTest.test_rack_constraint_repair (1/36 runs)
  failure at 2022-12-21T05:00:23.156Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4788#01853123-39d8-452b-b909-9c057db48f19

FAIL test: PartitionBalancerTest.test_rack_awareness (1/36 runs)
  failure at 2022-12-21T05:00:23.156Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4788#01853123-39d8-452b-b909-9c057db48f19

FAIL test: PartitionBalancerTest.test_decommission.kill_same_node=False.decommission_first=False (1/36 runs)
  failure at 2022-12-21T05:00:23.156Z: TimeoutError('')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4788#01853123-39d8-452b-b909-9c057db48f19

FAIL test: PartitionBalancerTest.test_fuzz_admin_ops (1/35 runs)
  failure at 2022-12-21T05:00:23.156Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4788#01853123-39d8-452b-b909-9c057db48f19

FAIL test: PartitionBalancerTest.test_unavailable_nodes (1/35 runs)
  failure at 2022-12-21T05:00:23.156Z: TimeoutError('failed to wait until status condition')
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4788#01853123-39d8-452b-b909-9c057db48f19