vortex icon indicating copy to clipboard operation
vortex copied to clipboard

feat: teach VarBinArray to only compute IsConstant when requested

Open danking opened this issue 1 year ago • 4 comments

I also remove the is_strict_sorted check because it provides litle value and is rather expensive, particularly now that we need not compute all statistics to know if a VarBinArray is constant.

danking avatar Oct 10 '24 19:10 danking

I'm not certain this is any faster, so let's look at the benchmarks first.

danking avatar Oct 10 '24 22:10 danking

yeah, I see itertools has annoying bugs regarding sized, i.e. you can't use them on dyn traits

robert3005 avatar Oct 10 '24 22:10 robert3005

@danking could be the case that the varbiniterator is the slow thing. It's very naive the way it's written and we likely need a better version of it

robert3005 avatar Oct 10 '24 22:10 robert3005

Clearly some benefit to compression time but decompress is impaired in a few cases. We don’t have enough samples yet to know how much variance there is.

danking avatar Oct 10 '24 22:10 danking

This is not clearly better than develop.

danking avatar Oct 15 '24 13:10 danking