tools icon indicating copy to clipboard operation
tools copied to clipboard

Improve distribution support in import benchmark and benchmark different container types

Open jaffee opened this issue 7 years ago • 1 comments

We need to make sure we can import data which gives us a variety of different container types in Pilosa's roaring format to test against.

The current import tool can probably reliably generate array and bitmap containers, but I imagine we'll need a way to generate RLE containers as well.

I'm picturing a spawn config which will run three imports (each into a different frame), one which will result in primarily array containers, one bitmap, and one RLE. Then we can have a variety of queries which combine rows from the different frames to test their relative performance.

jaffee avatar Aug 01 '17 18:08 jaffee

I started working on this for roaring tests, my hope was to devise a function randomBitset(Nbits, Nruns) which would generate something "nearly" uniform but with the exact Nbits and Nruns specified.

alanbernstein avatar Aug 01 '17 18:08 alanbernstein