arrow2
arrow2 copied to clipboard
Arrow2 memory usage too much in `lexsort_to_indices` than arrow
Which is found by : https://github.com/datafuselabs/datafuse/issues/1274
Seems we forget to add shrink_to_fit
in
https://github.com/jorgecarleitao/arrow2/blob/fa3c2ce2dd4419256de8283f885538b6b011e5d4/src/compute/sort/lex_sort.rs#L175-L179
and some other places.
Seems not related to shrink_to_fit
, because it already has shrink_to_fit
in indices_sorted_unstable_by
.
need to profile the memory to see the differ :)