texpresso icon indicating copy to clipboard operation
texpresso copied to clipboard

Need better control for parallelism

Open kvark opened this issue 2 years ago • 1 comments

It appears that, currently, if I want to contol the parallel compressions of multiple textures, the choice is either enable rayon or reimplement fn compress() myself. I think it could be done better. For example, compress could be implemented on top of a helper like this:

fn compress_block_iter() ->impl Iterator<Item = BlockIndex>;

This way, if I want rayon, I will just convert the regular iterator into sequential with something like into_par_iter().

The advantage here is the following:

  1. no dependency on Rayon needed
  2. can support other ways of parallel execution, such as Choir, without too much boilerplate on the user side

kvark avatar May 29 '23 06:05 kvark

The rayon solution is mostly meant as an "easy" API, but it did occur to me that the rayon-specific code might really belong into the cli rather than the library itself.

I would also like to make the API more ergonomic to use in a massively parallel manner (OpenCL / compute shaders). Not sure how well iterators specifically lend themselves to that directly but as long as there is a "compress 1 block" API, producing an iterator over all blocks should be a zero-cost operation so I think that is a safe way to approach it.

jansol avatar May 29 '23 09:05 jansol