opteryx
opteryx copied to clipboard
✨ constant column compression
When a column has all the same values, rather than pass around the repeated value, use the header (available via the Columns class) to record the column and it's value.
The expectation is that this will decrease the size of the data being passed around and be faster.
We see improvement in speed of actions like agg and select on narrower tables.
The projection, join and access to columns will need to be updated as well as something to find and compress the table.