quickwit
quickwit copied to clipboard
quickwit-oss
Add control plane first stone: a gRPC metastore service.
Description
Add a metastore service with 2 implementations:
- One backed by a
Metastoreinstance. - One backed by a gRPC client.
Important note
The PR still needs a lot of cleaning and testing but I think the API is good so ready to have some feedback about the API.
Is there a ticket that covers the design and reason for introducing the control plane service type and the plans for what it will solve? It looks like it's mirroring the metastore APIs, is it intended that nodes will talk to this service instead of talking to the metastore directly? Is it intended to be stateless? Will it be expected that you run more than one?
I know there's talk about having the merge planner run in this plane, what else is envisioned to live here?
@kstaken some short answers so that you know a bit what's going on there:
Is there a ticket that covers the design and reason for introducing the control plane service type and the plans for what it will solve?
yes I will document that very soon (hopefully by next week)
It looks like it's mirroring the metastore APIs, is it intended that nodes will talk to this service instead of talking to the metastore directly? Is it intended to be stateless? Will it be expected that you run more than one?
Yes, this is correct. For the moment the plan is to have indexers making calls to the control plane that reads/writes to the metastore. In the end, we will need to ensure the high availability of the control plane but that subject will be tackled later.
I know there's talk about having the merge planner run in this plane, what else is envisioned to live here?
We will run the merge planner there indeed. We will also handle indexing scheduling to decide which node should index a given (source_id, set of partition_ids). The solution should obviously work well with indexing with Kafka consumers.
There is an open issue #1208 but I need to add there the current plan.
Closing this old PR which is replaced by https://github.com/quickwit-oss/quickwit/pull/1853