burrow icon indicating copy to clipboard operation
burrow copied to clipboard

[Vent] Concurrent consumers to increase throughput

Open gregdhill opened this issue 6 years ago • 1 comments

Vent is currently limited to a single burrow instance, to improve throughput we need some way to multiplex over multiple remote burrow instances.

Read say 1000 blocks from node 0, then the 1000 blocks after those from node 1 and so on. You could have N goroutines, 1 for each node and then they could race each other to update a offset uint64 using sync/atomic to the current highest block not scheduled, so node 0 would atomically set offset to 1000, then whichever node is ready first (one of the others) would race to update it to 2000, and so on - then they would all stream into buffered chans in a priority queue. = @silasdavis

gregdhill avatar Jun 08 '19 16:06 gregdhill

This can also be extended to the burrow dump client. (see also: #1142)

gregdhill avatar Jun 15 '19 10:06 gregdhill