redpanda
redpanda copied to clipboard
redpanda-data
Shard placement table persistence
Add persistence to shard_placement_table. Now all shard placement updates are persisted in kvstore and restored at startup. Also implement migrating from old topic table-determined placement and update the stress test with migrations, restarts and post-restart checks.
At this point shard assignments are still ultimately determined by topic_table, but we'll already be able to better track ongoing x-shard transfers after the restart. Also, everything is set for transitioning to node-local placement decisions.
Backports Required
- [x] none - not a bug fix
- [ ] none - this is a backport
- [ ] none - issue does not exist in previous branches
- [ ] none - papercut/not impactful enough to backport
- [ ] v24.1.x
- [ ] v23.3.x
- [ ] v23.2.x
Release Notes
- none
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f536f-edc0-402f-9e52-2ca50e32f1a1:
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_delete_test.TopicDeleteTest.topic_delete_test.with_restart=True"
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f536f-edc7-45f1-a473-165a98b13ce2:
"rptest.tests.topic_delete_test.TopicDeleteTest.topic_delete_test.with_restart=False"
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f536f-edc2-43ef-9139-a303bdda8783:
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteStressTest.stress_test"
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f536f-edc5-40f5-ad6c-624808f78f3c:
"rptest.tests.topic_delete_test.TopicDeleteTest.topic_delete_orphan_files_test"
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-280d-4c80-bb7a-d2bb9e5ddb04:
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=True.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_unavailable_test.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteTest.topic_delete_test.with_restart=True"
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-2807-4819-b098-183bc81b69b5:
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=False.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_installed_snapshots_test"
"rptest.tests.topic_delete_test.TopicDeleteTest.topic_delete_orphan_files_test"
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-2804-42a2-9dde-2ffdbf6632a8:
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=True.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteStressTest.stress_test"
new failures in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-280a-4e25-9bbd-469cdfc10350:
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=False.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_unavailable_test.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_delete_test.TopicDeleteTest.topic_delete_test.with_restart=False"
new failures in https://buildkite.com/redpanda/redpanda/builds/48797#018f5531-11b1-411c-b5f0-fa728febde49:
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.ABS"
new failures in https://buildkite.com/redpanda/redpanda/builds/48797#018f5531-11b4-4a59-a548-e8a22eabd741:
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3"
new failures in https://buildkite.com/redpanda/redpanda/builds/48797#018f5538-91ba-481f-a721-ad4d2a0382d9:
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.S3"
new failures in https://buildkite.com/redpanda/redpanda/builds/48797#018f5538-91b8-463c-b80f-c0ab8e17b0a1:
"rptest.tests.e2e_shadow_indexing_test.EndToEndThrottlingTest.test_throttling.cloud_storage_type=CloudStorageType.ABS"
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/48784#018f5377-2807-4819-b098-183bc81b69b5
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/49101#018f777c-3dc7-4eaa-a252-0b3d846c7341
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/49101#018f777c-3dc4-4737-9422-9838579fbad7
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/49101#018f780d-974c-47ff-afa0-905c06a31a3d
Had to rebase due to kvstore changes in dev, actual changes are
- https://github.com/redpanda-data/redpanda/compare/611e589232fd96efb5b5f41ff925dbf3f5150bcb..9479497b653bb8d69c45bea86b966af70bacbcea
- and https://github.com/redpanda-data/redpanda/compare/e1c8fa261f9a686fda5582aa3a7025074b875168..faa76fd8f26ed39511e9b8c0999fcda55363dddf