subql icon indicating copy to clipboard operation
subql copied to clipboard

Parallel indexing

Open stwiname opened this issue 2 years ago • 0 comments

Description

Now that we support worker threads we can extend this functionality to index data in parallel. The majority of the work will need to be done around keeping track of the latest indexed block. We could use a similar approach as with https://github.com/subquery/go-subql-substrate-dictionary to achieve this.

Initially the feature can be put behind a flag, but in the future it would be great if we could be smart about parallelisation based on the users project and determining if a block is dependent on another blocks data.

stwiname avatar Jul 28 '22 22:07 stwiname