subql
subql copied to clipboard
Parallel indexing
Description
Now that we support worker threads we can extend this functionality to index data in parallel. The majority of the work will need to be done around keeping track of the latest indexed block. We could use a similar approach as with https://github.com/subquery/go-subql-substrate-dictionary to achieve this.
Initially the feature can be put behind a flag, but in the future it would be great if we could be smart about parallelisation based on the users project and determining if a block is dependent on another blocks data.