Trill icon indicating copy to clipboard operation
Trill copied to clipboard

Columnar data format efficiency: do not create unnecessary columns

Open cybertyche opened this issue 6 years ago • 0 comments

At present, the columnar data format creates columns for every field and property in the type. For instance, a struct with fields a, b, and c will yield batches that have arrays for a, b, and c as well. However, if no downstream operators require access to field b, it makes no sense to allocate any arrays for it let alone populate them. This change need not actually change the type of the generated columnar format, just that the arrays for unneeded fields don't get allocated or populated.

cybertyche avatar Dec 10 '18 00:12 cybertyche