redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

config/datalake: add options for datalake topics

Open jcipar opened this issue 1 year ago • 0 comments

This change adds two new configuration options for controlling datalake support as part of the Iceberg project. One option, at the cluster level, can be used to globally enable or disable datalake support. The topic option turns on datalake support for a specific topic. A topic is considered "datalake-enabled" when both options are set.

This change also adds logging in ntp_archiver_service to indicate the state of topics as they are uploaded.

These new config options are experimental, and should not be enabled outside of development experiments.

  • Cluster-level config: experimental_enable_datalake
  • Topic-level config: experimental_datalake_topic

Backports Required

  • [x] none - not a bug fix
  • [ ] none - this is a backport
  • [ ] none - issue does not exist in previous branches
  • [ ] none - papercut/not impactful enough to backport
  • [ ] v24.1.x
  • [ ] v23.3.x
  • [ ] v23.2.x

Release Notes

  • none

jcipar avatar May 15 '24 21:05 jcipar