marten icon indicating copy to clipboard operation
marten copied to clipboard

Projection Shard Work Distribution in the Async Daemon

Open jeremydmiller opened this issue 3 years ago • 0 comments

This is probably going to be prime for not making the 4.0 cut, but let's still talk about it.

This is dependent first on #1718 and #1715.

If we know all the possible running nodes active in the system, plus all the known projection shards running asynchronously, the leader process should be extended to try to distribute different shards across different running nodes. We'd need a system to detect new nodes coming up and old nodes going offline. Each node should be listening for assignment changes by effectively just polling a table.

jeremydmiller avatar Jan 23 '21 19:01 jeremydmiller