redpanda-data
datalake: introduce partition translator
A partition translator is responsible for translating data from a given partition periodically. A single instance of translator should be active for a given partition and is attached to the leader replica.
The translator periodically wakes up, checks if there is new data that is pending translation and creates a reader for the data range and hands it off to low level iceberg translator that converts the data.
The frequency is controlled via topic property redpanda.iceberg.translation.interval.ms.
Added a cluster fixture that wires up an e2e test with basic validation.
More tests are coming in future patches once we have a ducktape service running.
Backports Required
- [ ] none - not a bug fix
- [ ] none - this is a backport
- [x] none - issue does not exist in previous branches
- [ ] none - papercut/not impactful enough to backport
- [ ] v24.2.x
- [ ] v24.1.x
- [ ] v23.3.x
Release Notes
- none
new failures in https://buildkite.com/redpanda/redpanda/builds/55859#01925a91-0a93-45be-8ea1-96c62422c2f7:
"rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.ABS.retention_type=retention.ms"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_bootstrap_with_recovery"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_config_batches.num_messages=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.drop_lifecycle_marker_test.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.ABS.check_mode=check_manifest_and_segment_metadata"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.S3.check_mode=check_manifest_existence"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=start_offset"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention.cloud_storage_type=CloudStorageType.ABS"
new failures in https://buildkite.com/redpanda/redpanda/builds/55859#01925a91-0a96-4f22-9673-6aded422bdd3:
"rptest.tests.e2e_shadow_indexing_test.ShadowIndexingInfiniteRetentionTest.test_segments_not_deleted.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.S3.retention_type=retention.bytes"
"rptest.tests.consumer_group_recovery_test.ConsumerOffsetsRecoveryTest.test_consumer_offsets_partition_recovery"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.TSWithAlreadyCompactedTopic.test_initial_upload"
"rptest.tests.upgrade_test.UpgradeFromPriorFeatureVersionCloudStorageTest.test_rolling_upgrade.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.drop_lifecycle_marker_test.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.ABS.check_mode=check_manifest_existence"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.S3.check_mode=no_check"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.enterprise_features_license_test.EnterpriseFeaturesTest.test_get_enterprise.with_license=False"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_creation_test.CreateTopicsTest.test_no_log_bloat_when_recreating_existing_topics"
new failures in https://buildkite.com/redpanda/redpanda/builds/55859#01925a91-0a9b-49dd-9567-e4e557566ecb:
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention_application.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.ABS.retention_type=retention.bytes"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_basic_controller_snapshot_restore"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_config_batches.num_messages=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_from_cloud.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_prevent_recovery.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=-1.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=1.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cluster_config_test.ClusterConfigAzureSharedKey.test_live_shared_key_change.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"
new failures in https://buildkite.com/redpanda/redpanda/builds/55859#01925a91-0a98-43c9-a9f0-42cee2008478:
"rptest.tests.e2e_shadow_indexing_test.ShadowIndexingInfiniteRetentionTest.test_segments_not_deleted.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention_application.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_topic_recovery_retention_settings"
"rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.S3.retention_type=retention.ms"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_from_cloud.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.timequery_test.TimeQueryTest.test_timequery.cloud_storage=True.batch_cache=False.spillover=True"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_prevent_recovery.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=-1.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=1.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.ABS.check_mode=no_check"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"
"rptest.tests.describe_topics_test.DescribeTopicsTest.test_describe_topics_with_documentation_and_types"
"rptest.tests.enterprise_features_license_test.EnterpriseFeaturesTest.test_get_enterprise.with_license=True"
new failures in https://buildkite.com/redpanda/redpanda/builds/55859#01925a94-b31d-4b24-9402-7c355365c613:
"rptest.tests.e2e_shadow_indexing_test.ShadowIndexingInfiniteRetentionTest.test_segments_not_deleted.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.S3.retention_type=retention.bytes"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_bootstrap_with_recovery"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore.message_size=5000.num_messages=100000.recovery_overrides=.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_config_batches.num_messages=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=False.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_unavailable_test.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.shadow_indexing_compacted_topic_test.TSWithAlreadyCompactedTopic.test_initial_upload"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.drop_lifecycle_marker_test.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.ABS.check_mode=check_manifest_and_segment_metadata"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.S3.check_mode=check_manifest_existence"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_creation_test.CreateTopicsTest.test_no_log_bloat_when_recreating_existing_topics"
new failures in https://buildkite.com/redpanda/redpanda/builds/55859#01925a94-b31b-439a-8fe8-c35a5fbb2b74:
"rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.ABS.retention_type=retention.ms"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.retention.local.target.bytes.1024.redpanda.remote.write.True.redpanda.remote.read.True.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=None.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_basic_controller_snapshot_restore"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=False.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore.message_size=5000.num_messages=100000.recovery_overrides=.retention.local.target.bytes.1024.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.drop_lifecycle_marker_test.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_prevent_recovery.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.cluster_config_test.ClusterConfigAzureSharedKey.test_live_shared_key_change.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.describe_topics_test.DescribeTopicsTest.test_describe_topics_with_documentation_and_types"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"
"rptest.tests.enterprise_features_license_test.EnterpriseFeaturesTest.test_get_enterprise.with_license=True"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention.cloud_storage_type=CloudStorageType.ABS"
new failures in https://buildkite.com/redpanda/redpanda/builds/55859#01925a94-b320-499f-8041-bc35acdbcd3f:
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention_application.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.ABS.retention_type=retention.bytes"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_gc_entire_manifest.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_recover_after_delete_records"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_from_cloud.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore.message_size=5000.num_messages=100000.recovery_overrides=.retention.local.target.bytes.1024.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.retention.local.target.bytes.1024.redpanda.remote.write.True.redpanda.remote.read.True.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.partition_movement_test.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=True.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_prevent_recovery.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.ABS.check_mode=no_check"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=-1.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=1.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.enterprise_features_license_test.EnterpriseFeaturesTest.test_get_enterprise.with_license=False"
new failures in https://buildkite.com/redpanda/redpanda/builds/55859#01925a94-b31f-4730-b67c-e93cb62cc45b:
"rptest.tests.e2e_shadow_indexing_test.ShadowIndexingInfiniteRetentionTest.test_segments_not_deleted.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_size_based_retention_application.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_topic_recovery_retention_settings"
"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting_with_partition_moves"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_aborted_tx.recovery_overrides=.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.S3.retention_type=retention.ms"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.partition_movement_test.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_gc_entire_manifest.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=True.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore.message_size=5000.num_messages=100000.recovery_overrides=.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore_with_config_batches.num_messages=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_from_cloud.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.timequery_test.TimeQueryTest.test_timequery.cloud_storage=True.batch_cache=False.spillover=True"
"rptest.tests.upgrade_test.UpgradeFromPriorFeatureVersionCloudStorageTest.test_rolling_upgrade.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.ABS.check_mode=check_manifest_existence"
"rptest.tests.topic_recovery_test.TopicRecoveryTest.test_many_partitions.cloud_storage_type=CloudStorageType.S3.check_mode=no_check"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=-1.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.archival_test.ArchivalTest.test_timeboxed_uploads.acks=1.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=start_offset"
new failures in https://buildkite.com/redpanda/redpanda/builds/55874#01925eec-440f-4110-a630-2ddd65e5a063:
"rptest.tests.describe_topics_test.DescribeTopicsTest.test_describe_topics_with_documentation_and_types"
non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56471#01928bff-bf53-4108-af28-6f15076435e5:
"rptest.tests.alter_topic_configuration_test.AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node.incremental_update=False"
non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56494#01928d2e-96f7-4a29-b281-7d24b13deb18:
"rptest.tests.alter_topic_configuration_test.AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node.incremental_update=False"
non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56494#01928d43-4571-4c0c-b519-d72d37d7872f:
"rptest.tests.alter_topic_configuration_test.AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node.incremental_update=False"
"rptest.transactions.transactions_test.TxUpgradeRevertTest.test_snapshot_compatibility"
non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56516#01928f2f-7247-4933-a76f-d80c4db4de84:
"rptest.tests.alter_topic_configuration_test.AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node.incremental_update=False"
non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56516#01928f49-bf60-4b86-b90c-09524cd0400f:
"rptest.tests.alter_topic_configuration_test.AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node.incremental_update=False"
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/55859#01925a91-0a93-45be-8ea1-96c62422c2f7
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/55859#01925a91-0a96-4f22-9673-6aded422bdd3
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/55859#01925a91-0a9b-49dd-9567-e4e557566ecb
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/55859#01925a94-b31d-4b24-9402-7c355365c613
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/55859#01925a94-b320-499f-8041-bc35acdbcd3f
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/55859#01925a94-b31b-439a-8fe8-c35a5fbb2b74
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56471#01928bff-bf4f-4a16-b75b-c13fa9b82a06 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56471#01928bff-bf57-4ba2-9cae-90664d55e046 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56471#01928bff-bf59-4d45-9ba5-2d9d12e17d44 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56494#01928d2e-96fa-4c84-a875-c06ccfb1549d ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56494#01928d43-456e-4c85-a7e6-b15aabefebbc ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56494#01928d43-4573-48ba-bf9b-02c00650b986 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56516#01928f2f-724b-4c38-bd94-f1cc4342d6b6 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56516#01928f49-bf5e-428d-9e35-e2b61cddc1ca ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56516#01928f49-bf65-4858-9850-f799f975d785 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56516#01928f49-bf63-4744-abbc-9c4bd010c8f2 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56955#0192b150-d0d5-44c7-9570-6e48226279fc ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56955#0192b150-d0d3-44f4-af3d-f9d8ff11a73e ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56955#0192b153-f1e4-4e5c-86c5-2616a53ced82 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56955#0192b153-f1e6-4f8a-b8ee-d4daed0f0c8f ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56955#0192b153-f1e8-4f83-b919-2f0424636b27 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56980#0192b2ef-afe1-4205-9cf3-1e9d9fc70bbc ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56980#0192b2ef-afe0-4ef4-b5e3-ca417dbdb61f ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56980#0192b2f4-4b48-4f8f-b039-782ab4c31d2f ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56980#0192b2f4-4b4a-41db-b19e-a6fcf145de08
the below tests from https://buildkite.com/redpanda/redpanda/builds/56471#01928bbb-7d35-452e-b1f6-590ee059a83a have failed and will be retried
kafka_server_rpfixture
the below tests from https://buildkite.com/redpanda/redpanda/builds/56494#01928ce8-3fbf-440b-aafa-ce990a0f6dc8 have failed and will be retried
translator_test_rpfixture
the below tests from https://buildkite.com/redpanda/redpanda/builds/56980#0192b2af-976b-4ac5-ba1c-56a716b68042 have failed and will be retried
translator_test_rpfixture
Retry command for Build#56471
please wait until all jobs are finished before running the slash command
/ci-repeat 1
tests/rptest/tests/cloud_storage_timing_stress_test.py::CloudStorageTimingStressTest.test_cloud_storage@{"cleanup_policy":"delete"}
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":false}
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":true}
Retry command for Build#56494
please wait until all jobs are finished before running the slash command
/ci-repeat 1
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":true}
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":false}
tests/rptest/tests/cloud_storage_timing_stress_test.py::CloudStorageTimingStressTest.test_cloud_storage@{"cleanup_policy":"delete"}
tests/rptest/transactions/transactions_test.py::TxUpgradeRevertTest.test_snapshot_compatibility
Failures are unrelated to the PR
- bazel is broken because the writer/avro is not fully bazel friendly.
- test_alter_config_shadow_indexing_mixed_node - I think thats a test issue.
- test_snapshot_compatibility - is unrelated, I see it happening on other PRs too, I'll take a look at it separately.
Retry command for Build#56516
please wait until all jobs are finished before running the slash command
/ci-repeat 1
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":false}
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":true}
tests/rptest/tests/cloud_storage_timing_stress_test.py::CloudStorageTimingStressTest.test_cloud_storage@{"cleanup_policy":"delete"}
this commit doesn't compile
@dotnwat this is the part that requires arrow-bazel integration.
this commit doesn't compile
@dotnwat this is the part that requires arrow-bazel integration.
got it. is this pr blocked then on arrow-bazel, or do we want to remove something from the bazel build in this pr?
Retry command for Build#56955
please wait until all jobs are finished before running the slash command
/ci-repeat 1
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":true}
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":false}
Ducktape test failures are unrelated, I've split it out into a separate PR just to make sure I'm not breaking something else.
Retry command for Build#56980
please wait until all jobs are finished before running the slash command
/ci-repeat 1
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":true}
tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node@{"incremental_update":false}
/ci-repeat 1 dt-repeat 10 tests/rptest/tests/alter_topic_configuration_test.py::AlterConfigMixedNodeTest.test_alter_config_shadow_indexing_mixed_node
This commit
cmake: tidyup dependency graph
fails to build in bazel
src/v/datalake/translation/partition_translator.cc:15:10: fatal error: 'datalake/batching_parquet_writer.h' file not found
15 | #include "datalake/batching_parquet_writer.h"
This commit
cmake: tidyup dependency graph
fails to build in bazel
src/v/datalake/translation/partition_translator.cc:15:10: fatal error: 'datalake/batching_parquet_writer.h' file not found 15 | #include "datalake/batching_parquet_writer.h"
hmm.. I fixed it in the commit right after, split for reviewer's convenience.. I'll fix-up the commit history then.
fixed it in the commit right after, split for reviewer's convenience.. I'll fix-up the commit history then.
thanks, i appreciate the former. i guess git bisect appreciates the later :)