Declarative sharding streams

[AbstractiveNord]

Proposed change

For now, users of NATS Jetstream should manually setup subject partitioning for sharding streams. Please make declarative way to shard streams across clusters of NATS Jetstream. For example, if new server is bootstrapped, tagged and ready, it's good to have a control-plane tool to join that server into stream and start sharding data. Also, if you have planned disconnect of server, for maintenance example, control-plane should take care of it. Of course, there is some tasks for cluster leader, like remap subjects, track data paths, move data, may be balance disks consumption. Since every server may have tags, tags may be used for parametrize sharding. Like, if servers limited by network, disks, CPUs and memory, it's useful data for plan sharding.

Use case

When streams are dynamic and may grows very quickly, and data inside is important, creating sub streams manually or half-manually is painful. Much better have some built-in control-plane for this.

Contribution

Yes

[tpihl]

Hi,

This one adds some interesting requirements, at least for us. We rely on order between messages (to make sure any consumer will end up in the same state regardless of network or other dynamic behaviour). When manually creating partitions, we can make sure all messages where internal order is important end up in the same partition.

How would an auto-partition work in this scenario @AbstractiveNord

[AbstractiveNord]

Hi,

This one adds some interesting requirements, at least for us. We rely on order between messages (to make sure any consumer will end up in the same state regardless of network or other dynamic behaviour). When manually creating partitions, we can make sure all messages where internal order is important end up in the same partition.

How would an auto-partition work in this scenario @AbstractiveNord

Hi, I consider your case as advanced, but if your logic relies on subjects routing and partitioning, there is no problem. If rules of partitioning are set, control-plane should take care of data and ensure compliance with the provided logic. Please provide some examples for me being able to answer more specific.

[AbstractiveNord]

Update. It's probably worth to try implement streams autosharding based on collected metadata. If each node knows latency between each other, available resources such as CPU, RAM, disk capacity, streams may use their raft consensus to establish, which node contains which chunks of stream, keys/subjects range for fast lookup, maybe some indexes-like data, like bloom filter, etc. Highly similar to partitions in Kafka. By saying chunks, I mean current implementation of stream, which is single Raft group only. By allowing for streams to maintain multiple Raft groups automatically, streams become more powerful, allows to manage higher throughput, etc.