itertools icon indicating copy to clipboard operation
itertools copied to clipboard

Mutable "stripes" of data

Open BatmanAoD opened this issue 6 years ago • 4 comments

I asked about this in Rayon, but the functionality is not actually specific to parallelism, so I've copied my request here instead:

Is there a way to get "chunks"-like functionality but with striping/stepping instead of contiguous memory?

I.e., if you have some data in a slice, and every nth element of this slice is part of the same independent workload, you could split up the buffer for processing on multiple threads in such a way that the data for each thread would interleaved with the other threads' data.

This doesn't seem to be the semantics of interleave, windows, step_by, or any of the other functions I've found so far in the documentation.

I am told that such an access pattern is common in image processing.

BatmanAoD avatar Sep 06 '19 17:09 BatmanAoD

ndarray implements a relatively efficient way to do that — array views, and the striping is actually the same thing as arranging the data into rows and columns (either of those being the stripe).

If you say image processing, then I'm thinking of high performance and the general iterator trait might not be the best for that.

bluss avatar Sep 06 '19 19:09 bluss

Tying back to rayon, there's also the ndarray-parallel crate... (and isn't ndarray itself supposed to be getting rayon support?)

cuviper avatar Sep 06 '19 20:09 cuviper

Yes! I'm sorry, it hasn't had a release for a few months. 0.13 has rayon support directly.

bluss avatar Sep 06 '19 22:09 bluss

@bluss Yes, that sounds very much like what I was thinking of! I will check how close ndarray-parallel is to what I'm picturing.

BatmanAoD avatar Sep 16 '19 04:09 BatmanAoD