atomic-server icon indicating copy to clipboard operation
atomic-server copied to clipboard

Consider parallelizing sorting in `Db::query`

Open joepio opened this issue 2 years ago • 1 comments

When Db::query is performed, most of the time we'll get an index hit. These are fast. However, if the index does not exist yet (i.e. because this is the first time that specific QueryFilter is used), it has to be built during this query. That involves running a tpf query and sorting it. The tpf query is pretty fast, as it's indexed, but the sorting could take quite a while.

This is where it could help to have some degree of paralellization. Not sure how this would work, though, but let's use this issue to gather some ideas.

@AlexMikhalev may have some ideas already :)

joepio avatar Aug 29 '22 12:08 joepio

If we decide we stick with tokio, tokio runtime already has work-stealing workers: https://docs.rs/tokio/latest/tokio/attr.main.html#multi-threaded-runtime so we can split work per CPU core. Also, we may consider merge sort for parallelized sorting.

AlexMikhalev avatar Aug 29 '22 14:08 AlexMikhalev