lakeFS icon indicating copy to clipboard operation
lakeFS copied to clipboard

Make LakeFSInputFile (metadata reader) JOINable in (almost) linear time

Open arielshaqed opened this issue 3 years ago • 0 comments

Speed up metadata client JOIN operations by telling the driver how to JOIN metaranges in linear time. We will use this to transform garbage collector from a quadratic process to an (almost) linear process (O(n log n), in practice, due to cost of seeking).

Metaranges are sorted, so this should be possible by telling LakeFSInputFile to return a ComposableRecordReader.

arielshaqed avatar Sep 07 '22 10:09 arielshaqed