chronon icon indicating copy to clipboard operation
chronon copied to clipboard

[Algo] Two Stack + Hop and Sawtooth + two stack

Open cenhao opened this issue 1 year ago • 0 comments

Summary

Implementing variants of the two stack algorithms:

  • two stack with hopping
  • using two stack to replace the hopRangeCache in sawtooth Test result can be found here.

This PR is meant to be a POC, not intended to be merged.

A new interesting factor was observed: likely due to JVM's JIT, all algorithms ran faster in the second and third runs.

Result: https://docs.google.com/spreadsheets/d/1Mdh_0sv4H-6Ep6URjnFFI3sJ2oeXK88J4wBY32TwOUw/edit#gid=562347570

Perf tldr: with simple aggregation like average, two stacks + hop perform quite well. Replacing hop range cache with two stacks did not significantly improve perf.

With expensive aggregation like top k, two stack + hops still performs relatively well in smaller query set, but suffers from large query set.

image

Test Plan

  • [x] Added Unit Tests
  • [ ] Covered by existing CI
  • [ ] Integration tested

Checklist

  • [ ] Documentation update

Reviewers

@nikhilsimha

cenhao avatar Jun 05 '23 16:06 cenhao