Kamil Braun

Logs are gone, jenkins job was garbage collected, cannot investigate Please upload logs to GH whenever reporting an issue

Lucky, found another recent failure https://jenkins.scylladb.com/job/scylla-master/job/dtest-release/494/testReport/junit/replace_address_test/TestReplaceAddress/Run_Dtest_Parallel_Cloud_Machines___FullDtest___full_split007___test_replace_shutdown_node_use_host_id_rbo_disabled_/ reopening and attaching logs [1707866770783_replace_address_test.py TestReplaceAddress test_replace_shutdown_node[use_host_id-rbo_disabled].zip](https://github.com/scylladb/scylladb/files/14374924/1707866770783_replace_address_test.py.TestReplaceAddress.test_replace_shutdown_node.use_host_id-rbo_disabled.zip) [dtest-gw3.log](https://github.com/scylladb/scylladb/files/14374925/dtest-gw3.log)

Node 4 (replacing node) booted. ``` INFO 2024-02-13 23:24:07,532 [shard 0:main] init - Scylla version 5.5.0~dev-0.20240213.3d81138852ba initialization completed. ``` But node 3 (replaced node) tried to resurrect? ``` INFO 2024-02-13...

The wait started at ``` 23:23:45,180 985 ccm DEBUG cluster.py :762 | test_replace_shutdown_node[use_host_id-rbo_disabled]: node4: Starting scylla: args=['/jenkins/workspace/scylla-master/dtest-release/scylla/.dtest/dtest-_1dg7k7h/test/node4/bin/scylla', '--options-file', '/jenkins/workspace/scylla-master/dtest-release/scylla/.dtest/dtest-_1dg7k7h/test/node4/conf/scylla.yaml', '--log-to-stdout', '1', '--abort-on-seastar-bad-alloc', '--abort-on-lsa-bad-alloc', '1', '--abort-on-internal-error', '1', '--api-address', '127.0.67.4', '--smp', '2',...

Ah, node 1 is waiting for node 4 to enter NORMAL status, but it never observes it

Could it be yet another instance of the same root cause? https://github.com/scylladb/scylladb/issues/16902

Nodes not observing node 4 as entering NORMAL, could be also the reason why node 4 received a gossip event about node 3 (from some other node that thinks node...

> Could it be yet another instance of the same root cause? > https://github.com/scylladb/scylladb/issues/16902 And https://github.com/scylladb/scylladb/issues/16668

That last failure is with raft topology, but we don't know if the original failure was with raft topology, we lost the logs. (I'm asking every time -- please upload...

Logs for last failure: [1708610230795_schema_management_test.py TestSchemaManagement test_update_schema_while_node_is_killed[alter_table].zip](https://github.com/scylladb/scylladb/files/14397789/1708610230795_schema_management_test.py.TestSchemaManagement.test_update_schema_while_node_is_killed.alter_table.zip) [dtest-gw2.log](https://github.com/scylladb/scylladb/files/14397790/dtest-gw2.log)