tools
tools copied to clipboard
Improve distribution support in import benchmark and benchmark different container types
We need to make sure we can import data which gives us a variety of different container types in Pilosa's roaring format to test against.
The current import tool can probably reliably generate array and bitmap containers, but I imagine we'll need a way to generate RLE containers as well.
I'm picturing a spawn config which will run three imports (each into a different frame), one which will result in primarily array containers, one bitmap, and one RLE. Then we can have a variety of queries which combine rows from the different frames to test their relative performance.
I started working on this for roaring tests, my hope was to devise a function randomBitset(Nbits, Nruns)
which would generate something "nearly" uniform but with the exact Nbits and Nruns specified.