opteryx icon indicating copy to clipboard operation
opteryx copied to clipboard

✨ constant column compression

Open joocer opened this issue 2 years ago • 0 comments

When a column has all the same values, rather than pass around the repeated value, use the header (available via the Columns class) to record the column and it's value.

The expectation is that this will decrease the size of the data being passed around and be faster.

We see improvement in speed of actions like agg and select on narrower tables.

The projection, join and access to columns will need to be updated as well as something to find and compress the table.

joocer avatar Dec 27 '22 10:12 joocer