wildebeest icon indicating copy to clipboard operation
wildebeest copied to clipboard

Use multiprocessing as well as threading

Open gsganden opened this issue 4 years ago • 0 comments

Is your feature request related to a problem? Please describe.

wildebeest would be more useful for a wider range of problems if it could use multiprocessing as well as multithreading.

Describe the solution you'd like

First split into multiple processes, perhaps the number of cores available by default, and then into multiple threads within each core, a la https://github.com/rom1504/img2dataset. I believe this change would never substantially hurt speed and would substantially help in jobs that are somewhat CPU-constrained, e.g. when we are resizing images as we download them.

Describe alternatives you've considered

We could use async, but that would make it harder for users to plug in their own operations.

gsganden avatar Dec 21 '21 19:12 gsganden