GraphScope icon indicating copy to clipboard operation
GraphScope copied to clipboard

Published 20 hours ago • verified publisher icon alibaba

refactor(interactive): Refactor the `AbstractArrowFragmentLoader` for better concurrency

Open zhanglei1949 opened this issue 2 years ago • 1 comments

Currently, graph loading is done concurrently at the granularity of files. However, there is a significant size difference between files. We should perform concurrency at the granularity of RecordBatch.

zhanglei1949 avatar Dec 05 '23 05:12 zhanglei1949

Also, ODPSFragment can refactored to allow parallel stream reading, since it allows splitting table by rows.

zhanglei1949 avatar Jan 02 '24 07:01 zhanglei1949

Closed since already implemented.

zhanglei1949 avatar Aug 14 '24 06:08 zhanglei1949