proxima-platform
proxima-platform copied to clipboard
Enhancement: Bulk FileFormat should knows requested attributes during read path
Currently we have this signature in FileFormat
interface
Reader openReader(Path path, EntityDescriptor entity) throws IOException;
Which reads all attributes stored in path
and then it is filtered by BlogLogReader just for requested.
This solution can be suboptimal for some FileFormats (eq parquet), where we can use for example push down filters and read just block which contains requested attributes.
We should probably pass list of requested attributes directly into FileFormat
which allows to read file more effectively.