horaedb icon indicating copy to clipboard operation
horaedb copied to clipboard

feat: support horaedb submit compaction task to remote

Open LeslieKid opened this issue 1 year ago • 2 comments

Rationale

The subtask to support compaction offloading. See #1545

Detailed Changes

Compaction node support remote compaction service

  • Define CompactionServiceImpl to support compaction rpc service.

  • Introduce NodeType to distinguish compaction node and horaedb node. Enable the deployment of compaction node.

  • Impl compaction_client for horaedb node to access remote compaction node.

Horaedb node support compaction offload

  • Introduce compaction_mode in analytic engine's Config to determine whether exec compaction offload or not.
  • Define CompactionNodePicker trait, supporting get remote compaction node info.
  • Impl RemoteCompactionRunner, supporting pick remote node and pass compaction task to the node.
  • Add docs (e.g. example-cluster-n.toml) to explain how to deploy a cluster supporting compaction offload.

Test Plan

LeslieKid avatar Sep 03 '24 01:09 LeslieKid

It looks good to me. I just helped to sort out codes, mainly move the compaction client to runner crate.

I am checking it in my local, let's merge it after the double check!

Rachelint avatar Sep 30 '24 09:09 Rachelint

I am checking it in my local, let's merge it after the double check!

I check your commits in my local. It's indeed better to move compaction client to runner crate.

I think we can merge it.

LeslieKid avatar Sep 30 '24 10:09 LeslieKid

Thanks @LeslieKid

Rachelint avatar Oct 29 '24 02:10 Rachelint