marten
marten copied to clipboard
Projection Shard Work Distribution in the Async Daemon
This is probably going to be prime for not making the 4.0 cut, but let's still talk about it.
This is dependent first on #1718 and #1715.
If we know all the possible running nodes active in the system, plus all the known projection shards running asynchronously, the leader process should be extended to try to distribute different shards across different running nodes. We'd need a system to detect new nodes coming up and old nodes going offline. Each node should be listening for assignment changes by effectively just polling a table.