redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard
verified publisher icon redpanda-data

[CORE-14918] Cleanup Redpanda service on all preallocated test nodes

Open dotnwat opened this issue 1 month ago • 4 comments

This PR addresses an issue where preallocated test nodes may retain running Redpanda broker processes from previous test runs (we try to clean up on test shutdown, but we also proactively try to clean on test start-up since sometimes shutdown cleanup isn't reliable), causing port conflicts when the nodes are reused. The fix proactively cleans all preallocated nodes before allocation, regardless of their intended service, to ensure any leftover Redpanda processes are removed.

Key Changes:

  • Add cleanup logic in preallocated_nodes() to clean all nodes before allocation
  • Remove unused RemoteClusterNode Protocol type
  • Improve debugging by logging process state when socket conflicts are detected

Fixes https://redpandadata.atlassian.net/browse/CORE-14918

Backports Required

  • [x] none - not a bug fix
  • [ ] none - this is a backport
  • [ ] none - issue does not exist in previous branches
  • [ ] none - papercut/not impactful enough to backport
  • [ ] v25.3.x
  • [ ] v25.2.x
  • [ ] v25.1.x

Release Notes

  • none

dotnwat avatar Dec 06 '25 20:12 dotnwat

Retry command for Build#77481

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"node_add","cleanup_policy":"compact,delete","test_mode":"tiered_storage"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":false,"source_cluster_spec":{"cluster_type":"redpanda"},"starting_offset":"earliest"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingMetricsTests.test_link_metrics
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_timestamps_match@{"source_cluster_spec":{"cluster_type":"redpanda"},"timestamp_type":"CreateTime"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_with_restart
tests/rptest/tests/cloud_retention_test.py::CloudRetentionTest.test_gc_entire_manifest@{"cloud_storage_type":1}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_read_when_cache_smaller_than_segment_size
tests/rptest/tests/consumer_group_recovery_tool_test.py::ConsumerOffsetsRecoveryToolTest.test_consumer_offsets_partition_count_change
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_on_demand_rebalancing@{"partition_count":1}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_crashed_node@{"cloud_topic":false}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_working_node@{"cloud_topic":true,"delete_topic":false,"tick_interval":3600000}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_flipping_decommission_recommission@{"cloud_topic":true,"node_is_alive":true}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recommissioning_node@{"cloud_topic":false}
tests/rptest/tests/shard_placement_test.py::ShardPlacementTest.test_upgrade
tests/rptest/tests/follower_fetching_test.py::FollowerFetchingTest.test_basic_follower_fetching@{"fetch_from":"fetch-from-local"}
tests/rptest/tests/idempotency_test.py::IdempotencyWriteCachingTest.test_idempotent_producers_write_caching
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":false,"force_back":false,"replication_factor":1,"unclean_abort":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":false,"force_back":true,"replication_factor":3,"unclean_abort":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move_x_core@{"replication_factor":3,"unclean_abort":true}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact,delete","key_set_cardinality":100,"storage_compaction_key_map_memory_kb":10}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact","key_set_cardinality":1000,"storage_compaction_key_map_memory_kb":131072}
tests/rptest/tests/tiered_storage_pause_test.py::TestTieredStoragePause.test_safe_pause_resume@{"allow_gaps_cluster_level":true,"allow_gaps_topic_level":false}
tests/rptest/tests/services_self_test.py::KgoVerifierSelfTest.test_kgo_verifier_multi
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"off","cleanup_policy":"compact","test_mode":"tiered_storage_fast_moves"}
tests/rptest/tests/connection_rate_limit_test.py::ConnectionRateLimitTest.connection_rate_test
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":true,"source_cluster_spec":{"cluster_type":"redpanda"},"starting_offset":"earliest"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_auto_prefix_trimming@{"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"with_failures":false}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_read_when_chunk_api_disabled
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_timestamps_match@{"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"timestamp_type":"LogAppendTime"}
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_on_demand_rebalancing@{"partition_count":20}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_crashed_node@{"cloud_topic":true}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_learner_gap_metrics
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_working_node@{"cloud_topic":false,"delete_topic":false,"tick_interval":5000}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recommissioning_node@{"cloud_topic":true}
tests/rptest/tests/partition_movement_upgrade_test.py::PartitionMovementUpgradeTest.test_basic_upgrade
tests/rptest/tests/follower_fetching_test.py::FollowerFetchingTest.test_basic_follower_fetching@{"fetch_from":"fetch-from-tiered-storage"}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":true,"force_back":true,"replication_factor":3,"unclean_abort":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":true,"force_back":false,"replication_factor":1,"unclean_abort":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_forced_cancellation
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact,delete","key_set_cardinality":100,"storage_compaction_key_map_memory_kb":131072}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact","key_set_cardinality":1000,"storage_compaction_key_map_memory_kb":3}
tests/rptest/tests/tiered_storage_pause_test.py::TestTieredStoragePause.test_safe_pause_resume@{"allow_gaps_cluster_level":false,"allow_gaps_topic_level":null}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"off","cleanup_policy":"compact,delete","test_mode":"tiered_storage_fast_moves"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":false,"source_cluster_spec":{"cluster_type":"redpanda"},"starting_offset":"latest"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkTopicFailoverTests.test_link_failover@{"source_cluster_spec":{"cluster_type":"redpanda"},"with_failures":true}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_auto_prefix_trimming@{"source_cluster_spec":{"cluster_type":"redpanda"},"with_failures":false}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_timestamps_match@{"source_cluster_spec":{"cluster_type":"redpanda"},"timestamp_type":"LogAppendTime"}
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingManyPartitionsTest.test_many_partitions_recovery
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_adding_multiple_nodes_to_the_cluster@{"partition_count":1}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation@{"cloud_topic":false,"delete_topic":false}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_working_node@{"cloud_topic":true,"delete_topic":false,"tick_interval":5000}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_read_when_segment_size_smaller_than_chunk_size
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_multiple_decommissions@{"cloud_topic":false}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recommissioning_node_finishes@{"cloud_topic":false}
tests/rptest/tests/offset_for_leader_epoch_test.py::OffsetForLeaderEpochTest.test_offset_for_leader_epoch_transfer
tests/rptest/tests/follower_fetching_test.py::FollowerFetchingTest.test_follower_fetching_with_maintenance_mode@{"fetch_from":"fetch-from-cloud-topic"}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":false,"force_back":true,"replication_factor":1,"unclean_abort":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":false,"force_back":false,"replication_factor":3,"unclean_abort":true}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact,delete","key_set_cardinality":100,"storage_compaction_key_map_memory_kb":3}
tests/rptest/tests/log_compaction_test.py::LogCompactionTxRemovalTest.test_tx_control_batch_removal
tests/rptest/tests/tiered_storage_pause_test.py::TestTieredStoragePause.test_safe_pause_resume@{"allow_gaps_cluster_level":true,"allow_gaps_topic_level":null}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_prefetch_chunks@{"prefetch":3}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_with_truncated_topic
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":true,"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"starting_offset":"latest"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_basic@{"shuffle_leadership":true,"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"}}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_and_upgrade
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_rebalancing_node@{"shutdown_decommissioned":false}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_flipping_decommission_recommission@{"cloud_topic":false,"node_is_alive":false}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recycle_all_nodes@{"auto_assign_node_id":true}
tests/rptest/tests/shard_placement_test.py::ShardPlacementTest.test_manual_rebalance
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move_x_core@{"replication_factor":1,"unclean_abort":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":true,"force_back":true,"replication_factor":1,"unclean_abort":true}
tests/rptest/tests/shadow_linking_rnot_test.py::ShadowLinkingRandomOpsTest.test_node_operations@{"failures":true}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_fallback_mode
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":true,"source_cluster_spec":{"cluster_type":"redpanda"},"starting_offset":"timestamp"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":true,"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"starting_offset":"earliest"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_with_large_msgs
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_adding_nodes_to_cluster@{"partition_count":1}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_basic@{"shuffle_leadership":false,"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"}}
tests/rptest/tests/log_compaction_test.py::LogCompactionTxRemovalUpgradeTest.test_tx_control_batch_removal_with_upgrade@{"test_case_name":"Mixed aborts and commits"}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation@{"cloud_topic":true,"delete_topic":true}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_working_node@{"cloud_topic":false,"delete_topic":true,"tick_interval":5000}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recommissioning_one_of_decommissioned_nodes@{"cloud_topic":true}
tests/rptest/tests/redpanda_multi_region_test.py::RedpandaNetworkDelayTest.test_basic_traffic_shaping
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":true,"force_back":false,"replication_factor":1,"unclean_abort":true}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":true,"force_back":true,"replication_factor":3,"unclean_abort":true}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":false,"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"starting_offset":"latest"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkTopicFailoverTests.test_link_topic_failover@{"source_cluster_spec":{"cluster_type":"redpanda"},"with_failures":true}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_basic@{"shuffle_leadership":false,"source_cluster_spec":{"cluster_type":"redpanda"}}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_with_transactions
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_prefetch_chunks@{"prefetch":0}
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_adding_nodes_to_cluster@{"partition_count":20}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_node_rf_1_replica
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_working_node@{"cloud_topic":true,"delete_topic":true,"tick_interval":5000}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recycle_all_nodes@{"auto_assign_node_id":false}
tests/rptest/tests/shard_placement_test.py::ShardPlacementTest.test_core_count_change
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":false,"force_back":true,"replication_factor":1,"unclean_abort":true}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move_node_down
tests/rptest/tests/log_compaction_test.py::LogCompactionTxRemovalUpgradeTest.test_tx_control_batch_removal_with_upgrade@{"test_case_name":"Multi-segment spanning transactions"}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"off","cleanup_policy":"compact,delete","test_mode":"tiered_storage"}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"node_add","cleanup_policy":"compact,delete","test_mode":"tiered_storage_fast_moves"}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancellations_interrupted_with_restarts@{"replication_factor":1}
tests/rptest/tests/cloud_retention_test.py::CloudRetentionTest.test_cloud_retention@{"cloud_storage_type":1,"max_consume_rate_mb":20}
tests/rptest/tests/idempotency_stress_test.py::IdempotencyStressTest.producer_id_stress_test@{"max_producer_ids":1000}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact","key_set_cardinality":100,"storage_compaction_key_map_memory_kb":10}
tests/rptest/tests/tiered_storage_enable_test.py::TestEnablingTieredStorage.test_enabling_tiered_storage_on_old_topic
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkTopicFailoverTests.test_link_failover@{"source_cluster_spec":{"cluster_type":"redpanda"},"with_failures":false}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":true,"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"starting_offset":"timestamp"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkTopicFailoverTests.test_link_failover@{"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"with_failures":false}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_timestamps_match@{"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"timestamp_type":"CreateTime"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_topic_delete@{"source_cluster_spec":{"cluster_type":"redpanda"}}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_read_chunks
tests/rptest/tests/compatibility/arroyo_test.py::ArroyoTest.test_arroyo_test_suite
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_cancel_ongoing_movements@{"cloud_topic":true}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_working_node@{"cloud_topic":false,"delete_topic":false,"tick_interval":3600000}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_flipping_decommission_recommission@{"cloud_topic":false,"node_is_alive":true}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recommissioning_do_not_stop_all_moves_node@{"cloud_topic":true}
tests/rptest/tests/shard_placement_test.py::ShardPlacementTest.test_node_join@{"disable_license":true}
tests/rptest/tests/follower_fetching_test.py::FollowerFetchingTest.test_basic_follower_fetching@{"fetch_from":"fetch-from-cloud-topic"}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_all_moves_in_cluster
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":true,"force_back":false,"replication_factor":3,"unclean_abort":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move_x_core@{"replication_factor":3,"unclean_abort":false}
tests/rptest/tests/consumer_group_recovery_test.py::ConsumerOffsetsRecoveryTest.test_consumer_offsets_partition_recovery@{"force_offset_upload_failures":true}
tests/rptest/tests/upgrade_test.py::UpgradeBackToBackTest.test_upgrade_with_all_workloads@{"single_upgrade":true}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"off","cleanup_policy":"compact","test_mode":"tiered_storage"}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"node_add","cleanup_policy":"compact","test_mode":"tiered_storage_fast_moves"}
tests/rptest/tests/cloud_retention_test.py::CloudRetentionTest.test_cloud_retention@{"cloud_storage_type":2,"max_consume_rate_mb":20}
tests/rptest/tests/idempotency_stress_test.py::IdempotencyStressTest.producer_id_stress_test@{"max_producer_ids":100}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact,delete","key_set_cardinality":1000,"storage_compaction_key_map_memory_kb":3}
tests/rptest/tests/shadow_linking_rnot_test.py::ShadowLinkingRandomOpsTest.test_node_operations@{"failures":false}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"off","cleanup_policy":"compact,delete","test_mode":"no_tiered_storage"}
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingTrafficShapingTest.test_many_partitions_recovery_with_traffic_shaping
tests/rptest/tests/random_node_operations_smoke_test.py::RedpandaNodeOperationsSmokeTest.test_node_ops_smoke_test@{"cloud_storage_type":1,"mixed_versions":true}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":false,"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"starting_offset":"earliest"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":false,"source_cluster_spec":{"cluster_type":"redpanda"},"starting_offset":"timestamp"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkTopicFailoverTests.test_link_topic_failover@{"source_cluster_spec":{"cluster_type":"redpanda"},"with_failures":false}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_with_failures
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_auto_prefix_trimming@{"source_cluster_spec":{"cluster_type":"redpanda"},"with_failures":true}
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_adding_node_with_unavailable_node
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation@{"cloud_topic":false,"delete_topic":true}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recommissioning_one_of_decommissioned_nodes@{"cloud_topic":false}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_working_node@{"cloud_topic":true,"delete_topic":true,"tick_interval":3600000}
tests/rptest/tests/follower_fetching_test.py::FollowerFetchingTest.test_follower_fetching_with_maintenance_mode@{"fetch_from":"fetch-from-tiered-storage"}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":false,"force_back":false,"replication_factor":1,"unclean_abort":true}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":false,"force_back":true,"replication_factor":3,"unclean_abort":true}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact,delete","key_set_cardinality":1000,"storage_compaction_key_map_memory_kb":131072}
tests/rptest/tests/log_compaction_test.py::LogCompactionTxRemovalUpgradeTest.test_tx_control_batch_removal_with_upgrade@{"test_case_name":"All commits"}
tests/rptest/tests/node_pool_migration_test.py::DisableTieredStorageTest.test_disable_tiered_storage@{"disable_mode":"pause uploads"}
tests/rptest/tests/tiered_storage_pause_test.py::TestTieredStoragePause.test_safe_pause_resume@{"allow_gaps_cluster_level":true,"allow_gaps_topic_level":true}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"node_add","cleanup_policy":"compact","test_mode":"tiered_storage"}
tests/rptest/tests/cloud_retention_test.py::CloudRetentionTest.test_gc_entire_manifest@{"cloud_storage_type":2}
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_moves_with_local_retention@{"use_topic_property":true}
tests/rptest/tests/log_compaction_test.py::LogCompactionSchedulingTest.dirty_ratio_scheduling_test@{"key_set_cardinality":1000}
tests/rptest/tests/follower_fetching_test.py::IncrementalFollowerFetchingTest.test_incremental_fetch_from_follower@{"follower_offline":true}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact","key_set_cardinality":1000,"storage_compaction_key_map_memory_kb":10}
tests/rptest/tests/services_self_test.py::KgoVerifierSelfTest.test_kgo_verifier
tests/rptest/tests/tiered_storage_pause_test.py::TestTieredStoragePause.test_safe_pause_resume@{"allow_gaps_cluster_level":false,"allow_gaps_topic_level":false}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"node_add","cleanup_policy":"compact","test_mode":"no_tiered_storage"}
tests/rptest/tests/cloud_retention_test.py::CloudRetentionTest.test_cloud_retention@{"cloud_storage_type":2,"max_consume_rate_mb":null}
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_fast_node_addition
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancellations_interrupted_with_restarts@{"replication_factor":3}
tests/rptest/tests/idempotency_stress_test.py::IdempotencyStressTest.producer_id_stress_test@{"max_producer_ids":3000}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact","key_set_cardinality":100,"storage_compaction_key_map_memory_kb":131072}
tests/rptest/tests/tiered_storage_pause_test.py::TestTieredStoragePause.test_resume@{"allow_gaps_cluster_level":false,"allow_gaps_topic_level":null}
tests/rptest/tests/cloud_storage_chunk_read_path_test.py::CloudStorageChunkReadTest.test_prefetch_chunks@{"prefetch":5}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":false,"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"starting_offset":"timestamp"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_basic@{"shuffle_leadership":true,"source_cluster_spec":{"cluster_type":"redpanda"}}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_topic_delete@{"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"}}
tests/rptest/tests/consumer_group_recovery_test.py::ConsumerOffsetsRecoveryTest.test_consumer_offsets_partition_recovery@{"force_offset_upload_failures":false}
tests/rptest/tests/consumer_offsets_consistency_test.py::ConsumerOffsetsConsistencyTest.test_flipping_leadership
tests/rptest/tests/idempotency_test.py::IdempotencySnapshotDelivery.test_recovery_after_snapshot_is_delivered
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_cancel_ongoing_movements@{"cloud_topic":false}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_rebalancing_node@{"shutdown_decommissioned":true}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_flipping_decommission_recommission@{"cloud_topic":true,"node_is_alive":false}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recommissioning_do_not_stop_all_moves_node@{"cloud_topic":false}
tests/rptest/tests/shard_placement_test.py::ShardPlacementTest.test_node_join@{"disable_license":false}
tests/rptest/tests/upgrade_test.py::UpgradeBackToBackTest.test_upgrade_with_all_workloads@{"single_upgrade":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_all_moves_from_node
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":false,"force_back":false,"replication_factor":3,"unclean_abort":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move_x_core@{"replication_factor":1,"unclean_abort":true}
tests/rptest/tests/random_node_operations_smoke_test.py::RedpandaNodeOperationsSmokeTest.test_node_ops_smoke_test@{"cloud_storage_type":1,"mixed_versions":false}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkCustomStartOffsetSelectionTests.test_starting_offset@{"failures":true,"source_cluster_spec":{"cluster_type":"redpanda"},"starting_offset":"latest"}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkTopicFailoverTests.test_link_topic_failover@{"source_cluster_spec":{"cluster_type":"kafka","kafka_quorum":"COMBINED_KRAFT","kafka_version":"3.8.0"},"with_failures":false}
tests/rptest/tests/cluster_linking_e2e_test.py::ShadowLinkingReplicationTests.test_replication_with_compaction
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_adding_multiple_nodes_to_the_cluster@{"partition_count":20}
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_topic_hot_spots
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation@{"cloud_topic":true,"delete_topic":false}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_working_node@{"cloud_topic":false,"delete_topic":true,"tick_interval":3600000}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_recommissioning_node_finishes@{"cloud_topic":true}
tests/rptest/tests/follower_fetching_test.py::FollowerFetchingTest.test_follower_fetching_with_maintenance_mode@{"fetch_from":"fetch-from-local"}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":true,"force_back":true,"replication_factor":1,"unclean_abort":false}
tests/rptest/tests/partition_move_interruption_test.py::PartitionMoveInterruption.test_cancelling_partition_move@{"compacted":true,"force_back":false,"replication_factor":3,"unclean_abort":true}
tests/rptest/tests/node_pool_migration_test.py::DisableTieredStorageTest.test_disable_tiered_storage@{"disable_mode":"disable tiered storage"}
tests/rptest/tests/log_compaction_test.py::LogCompactionTxRemovalUpgradeTest.test_tx_control_batch_removal_with_upgrade@{"test_case_name":"All aborts"}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"node_add","cleanup_policy":"compact,delete","test_mode":"no_tiered_storage"}
tests/rptest/tests/cloud_retention_test.py::CloudRetentionTest.test_cloud_retention@{"cloud_storage_type":1,"max_consume_rate_mb":null}
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_moves_with_local_retention@{"use_topic_property":false}
tests/rptest/tests/follower_fetching_test.py::IncrementalFollowerFetchingTest.test_incremental_fetch_from_follower@{"follower_offline":false}
tests/rptest/tests/services_self_test.py::KgoVerifierMultiNodeSelfTest.test_kgo_verifier_multi_node
tests/rptest/tests/log_compaction_test.py::LogCompactionSchedulingTest.dirty_ratio_scheduling_test@{"key_set_cardinality":100}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact","key_set_cardinality":100,"storage_compaction_key_map_memory_kb":3}
tests/rptest/tests/tiered_storage_pause_test.py::TestTieredStoragePause.test_resume@{"allow_gaps_cluster_level":true,"allow_gaps_topic_level":null}
tests/rptest/tests/node_pool_migration_test.py::NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool@{"balancing_mode":"off","cleanup_policy":"compact","test_mode":"no_tiered_storage"}
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingManyPartitionsTest.test_many_partitions_shutdown
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_multiple_decommissions@{"cloud_topic":true}
tests/rptest/tests/log_compaction_test.py::LogCompactionTest.compaction_stress_test@{"cleanup_policy":"compact,delete","key_set_cardinality":1000,"storage_compaction_key_map_memory_kb":10}
tests/rptest/tests/node_folder_deletion_test.py::NodeFolderDeletionTest.test_deleting_node_folder
tests/rptest/tests/tiered_storage_pause_test.py::TestTieredStoragePause.test_safe_pause_resume@{"allow_gaps_cluster_level":false,"allow_gaps_topic_level":true}

vbotbuildovich avatar Dec 06 '25 21:12 vbotbuildovich

Retry command for Build#77483

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_decommissioning_rebalancing_node@{"shutdown_decommissioned":false}

vbotbuildovich avatar Dec 07 '25 01:12 vbotbuildovich

CI test results

test results on build#77483
test_class test_method test_arguments test_kind job_url test_status passed reason test_history
EndToEndCloudTopicsTest test_delete_records null integration https://buildkite.com/redpanda/redpanda/builds/77483#019af612-1d4a-4b38-bcec-a4017e8a2b2f FLAKY 30/31 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0105, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.0312, p1=0.3868, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=EndToEndCloudTopicsTest&test_method=test_delete_records
NodesDecommissioningTest test_decommissioning_rebalancing_node {"shutdown_decommissioned": false} integration https://buildkite.com/redpanda/redpanda/builds/77483#019af60c-396d-430f-85cb-291900c2065e FLAKY 22/31 Test FAILS after retries.Significant increase in flaky rate(baseline=0.0904, p0=0.0042, reject_threshold=0.0100) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodesDecommissioningTest&test_method=test_decommissioning_rebalancing_node
NodesDecommissioningTest test_decommissioning_rebalancing_node {"shutdown_decommissioned": true} integration https://buildkite.com/redpanda/redpanda/builds/77483#019af60c-396e-4d5b-8993-d0b8cb3de5ba FLAKY 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0904, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.2473, p1=0.0584, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodesDecommissioningTest&test_method=test_decommissioning_rebalancing_node
PartitionReassignmentsTest test_reassignments_cancel null integration https://buildkite.com/redpanda/redpanda/builds/77483#019af612-1d4d-4e91-981f-440ad99011fc FLAKY 9/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0655, p0=0.4920, reject_threshold=0.0100. adj_baseline=0.1838, p1=0.4265, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=PartitionReassignmentsTest&test_method=test_reassignments_cancel
test results on build#77494
test_class test_method test_arguments test_kind job_url test_status passed reason test_history
NodesDecommissioningTest test_decommissioning_rebalancing_node {"shutdown_decommissioned": true} integration https://buildkite.com/redpanda/redpanda/builds/77494#019af6f2-08cc-4368-a6f4-d4f6ba66aede FAIL 0/1 https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodesDecommissioningTest&test_method=test_decommissioning_rebalancing_node
PartitionReassignmentsTest test_reassignments_cancel null integration https://buildkite.com/redpanda/redpanda/builds/77494#019af6f8-8f63-49c6-a7ff-b80f207ce031 FLAKY 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0440, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1263, p1=0.2592, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=PartitionReassignmentsTest&test_method=test_reassignments_cancel
test results on build#77501
test_class test_method test_arguments test_kind job_url test_status passed reason test_history
WriteCachingFailureInjectionE2ETest test_crash_all {"use_transactions": false} integration https://buildkite.com/redpanda/redpanda/builds/77501#019af9bd-6acc-4946-8c04-e49dd5565746 FLAKY 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0855, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.2351, p1=0.0685, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=WriteCachingFailureInjectionE2ETest&test_method=test_crash_all

vbotbuildovich avatar Dec 07 '25 01:12 vbotbuildovich

/dt

dotnwat avatar Dec 07 '25 03:12 dotnwat