venice icon indicating copy to clipboard operation
venice copied to clipboard

[controller] Support dangling topic check and deletion in topic cleanup service.

Open haoxu07 opened this issue 1 year ago • 0 comments

Summary, imperative, start upper case, don't end with a period

Add dangling topic checking functionality, TopicCleanupService will compare topics listed from kafka and topics from other type of pub sub client, if there is topic could not be found by kafka client but can be founded by other type of pub sub client, topic will be deleted by such cases:

  1. Real-time topic, if store is not hybrid, that topic will be deleted.
  2. Version topic, if store does not contain that version, the topic will be deleted.
  3. If there is no store available for that topic, the topic will be deleted.

Add a new config to enable this feature: CONTROLLER_DANGLING_TOPIC_CLEAN_UP_INTERVAL_SECOND. If its value is >0 and admin client class is not kafka, the feature will be turned on. Otherwise, it will be disabled.

How was this PR tested?

CI, integration test will be in internal repo with other type of pub sub system.

Does this PR introduce any user-facing changes?

  • [*] No. You can skip the rest of this section.
  • [] Yes. Make sure to explain your proposed changes and call out the behavior change.

haoxu07 avatar May 01 '24 01:05 haoxu07