redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

Purge unused topics after an expiry time

Open mattschumpert opened this issue 1 year ago • 1 comments

Similar to inactive consumer groups, we should have the ability to auto-purge topics from Redpanda after a period of complete inactivity (no Produce or Consume traffic), when the topics are assumed abandoned.

This helps in the situation that Kafka ecosystem tools do large amounts of spurious topic creation, in which case their partitions add unnecessary load to Redpanda. In some cases these are unused wasted resources creating drag on performance.

A global property such as

redpanda.topic.autodelete=60days

(off/unset by default), which deletes topics after the inactive period can help..

There should also be topic-level overrides to exempt a topic from auto-deletion if desired (topic remains forever despite inactivity)

JIRA Link: CORE-1327

mattschumpert avatar Jun 02 '23 22:06 mattschumpert