Data silos

[thedodd]

In GitLab by @doddzilla on Nov 20, 2020, 08:09

abstract

Our initial implementation will only use a single Raft group for managing data. However, we may explore the use of multiple smaller Raft groups to shard data. Alternatively, we may continue to use a single Raft group, but the create a data silo system where some nodes participate in the cluster only as data silos. The core Raft group would carve up streams once they get to a specific size, and will shard data across the silos. This will help to keep consensus overhead low, while also providing data sharding and scale.