h5spark icon indicating copy to clipboard operation
h5spark copied to clipboard

parallelize along user specified dimension

Open valiantljk opened this issue 8 years ago • 0 comments

Currently, H5Spark parallelize the IO along the slowest dimension, i.e., the dimension that changes slowest on disks. For example, for a 2D C array x[10][200], the h5spark will choose the first dimension to partition, and then the maximum partition it can have is only 10, which is also the maximum degree of parallelism.

If we want to parallelize along any user-specified dimension, the current code needs a little bit modification.

valiantljk avatar May 20 '16 17:05 valiantljk