ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Fix returning sparse column to Python layer after processing pipeline

Open alexowens90 opened this issue 11 months ago • 0 comments

https://github.com/man-group/ArcticDB/pull/1399 adds the ability to filter and project on sparse columns, tested in C++, and earlier merges allow grouping and aggregating on sparse columns too. However, copy_frame_data_to_buffer (the final stage of the processing pipeline) does not handle sparse columns correctly, unlike direct reads, which uses decode_or_expand_impl, which checks for the presence of a sparse map, and calls expand_dense_buffer_using_bitmap in this case. We should be able to add tests for this from the Python layer using the sparsify_floats kwarg to write to generate sparse data. Before merging, testing should be performed on a real library, see https://github.com/man-group/arcticdb-man/issues/82 for details.

alexowens90 avatar Mar 08 '24 10:03 alexowens90