storm icon indicating copy to clipboard operation
storm copied to clipboard

Better documentation of speed characteristics and index usage for `Find` vs `Select`

Open dimroc opened this issue 6 years ago • 3 comments

We've been writing some queries to later realize that Select and Query don't use indices.

https://github.com/asdine/storm/blob/dbd37722730b6cb703b5bd825c3f142d87358525/query.go#L9-L10

If that's correct, we have two scenarios that I'm hoping can leverage indices:

  1. We want to Find by a particular index, but OrderBy another. Is there a way to do this that leverages indices? We used to do the following by are moving to Find and sorting ourselves.
err := orm.Select(q.Eq("JobID", jobID)).OrderBy("CreatedAt").Reverse().Find(&runs) // doesn't use index

moving to

_ := orm.Find("JobID", jobID, &runs) // Use Find to leverage db index
sort.Sort(jobRunSorterAscending(runs))
  1. Find ing from a field with multiple matches (aka In). Would it be more perfomant for us to have multiple Finds? Seems unlikely.
err := orm.Select(q.In("Status", statuses)).Find(&runs) // doesn't use index

Thanks for any insight. We are considering a PR with this functionality if there's consensus.

dimroc avatar Jul 31 '18 20:07 dimroc

There is currently no index support in the Select method. While it could be a great addition, it is complicated to add them with the current code, so i decided to focus on the Storm v3 rewrite #211.

If you are interested in contributing by adding the support for indexes to Select, i'd be more than interested to give any insight you need 👍

asdine avatar Aug 02 '18 19:08 asdine

I think we will be. Storm and bolt are critical pieces of our app so we would love to be involved in the discussion at https://github.com/asdine/storm/issues/211.

As for a concise takeaway regarding this issue, is it safe to say Find gives us O(log(N)) and Select is O(N) and to just deal with that for v2?

dimroc avatar Aug 02 '18 19:08 dimroc

I think we will be. Storm and bolt are critical pieces of our app so we would love to be involved in the discussion at #211.

Actually i was proposing to add the support for indexes for v2 if you are interested (i'm glad you are interested in v3 though 👍 ) v3 is a complete rewrite and needs a lot of time to be ready, so i don't have time to focus on big features additions to v2, but i'm open to contributions.

Find gives us O(log(N)) and Select is O(N)

Yes exactly.

asdine avatar Aug 02 '18 20:08 asdine