ArcticDB
ArcticDB copied to clipboard
Fix returning sparse column to Python layer after processing pipeline
https://github.com/man-group/ArcticDB/pull/1399 adds the ability to filter and project on sparse columns, tested in C++, and earlier merges allow grouping and aggregating on sparse columns too.
However, copy_frame_data_to_buffer
(the final stage of the processing pipeline) does not handle sparse columns correctly, unlike direct reads, which uses decode_or_expand_impl
, which checks for the presence of a sparse map, and calls expand_dense_buffer_using_bitmap
in this case.
We should be able to add tests for this from the Python layer using the sparsify_floats
kwarg to write
to generate sparse data.
Before merging, testing should be performed on a real library, see https://github.com/man-group/arcticdb-man/issues/82 for details.