datajoint-matlab icon indicating copy to clipboard operation
datajoint-matlab copied to clipboard

Parallelize file transfer for external blobs

Open shenshan opened this issue 3 years ago • 1 comments

Feature Request

Problem

Fetching from an internal blob field and from external blob fields have big difference in performance.

Same to this issue on the python site: https://github.com/datajoint/datajoint-python/issues/806

Requirements

We should consider parallelizing the file downloads from external storage.

Benchmark

Here is a benchmark plotting showing the performance difference. This is for princeton database, running on a computer in the lab. The external storage on a shared drive in the institute.

image

shenshan avatar May 06 '21 21:05 shenshan

Screen Shot 2021-06-04 at 12 08 06 PM Screen Shot 2021-06-04 at 11 38 29 AM

shenshan avatar Jun 07 '21 21:06 shenshan