clickhouse-operator icon indicating copy to clipboard operation
clickhouse-operator copied to clipboard

Add ClickhouseBackup Plugin Support

Open jitendra1411 opened this issue 4 months ago • 2 comments

Overview

This PR introduces a new Backup plugin architecture to the ClickHouse Operator, enabling automated backup operations via an external plugin service. Issue: #1795


Current PR Includes

1. Operator Enhancements

  • CRDs

    • ClickhouseBackup (CHB) — manage one-time backup requests with S3 configuration, whitelist/blacklist filtering, and method/plugin settings.
  • Controller Updates

    • Operator reconciles CHB/CHSB CRs and communicates with a gRPC backup plugin using JSON-serialized request data.
    • Adds plugin discovery via CR labels/annotations.
    • Implements lifecycle management and status reporting.

2. Go Plugin Skeleton

  • Introduces a gRPC interface for backup operations using BackupRequest and BackupResult messages.
  • Plugin architecture allows future extension for operations like restore, encryption, incremental backups, etc.

Motivation & Benefits

  • Modular Design: Backup behavior is decoupled from the operator — update, replace, or extend the plugin without touching core operator logic.
  • Extensibility: Support for diverse backup implementations (e.g., S3, local disk, incremental, etc.).
  • Production-Ready: Enables defined workflows for both one-off and scheduled backups, ensuring reliable defense against data loss.

TODO

  • Restore Workflow: Implement a corresponding restore service via plugin.
  • Test Cases: Implement the test cases
  • ClickhouseScheduledBackup (CHSB) Implement code for scheduled backup
  • Generic gRPC Client: Implement a reusable gRPC client inside the operator for communication with different plugins.

Testing Notes

  • Validate CHB and CHSB workflows manually using a mock plugin server.
  • Status updates should reflect backup success or failure states.
  • Scheduling logic (cron parsing, suspension, etc.) requires validation.

jitendra1411 avatar Aug 21 '25 06:08 jitendra1411

Will this get accepted.

manishrawat1992 avatar Sep 08 '25 15:09 manishrawat1992

Would be really cool to see this getting merged

discostur avatar Nov 06 '25 16:11 discostur