cwlkr
Multiprocessing
Hello, I'm glad to see your project. It's useful for what I am doing now. But I have a problem to inquire you. I have put this WSI Normalizer in my dataloader, and the transform part is in the function getitem(). When I set the 'num_workers': > 1 to accelerate. It will cause the error like this:
I spent two days on it. However, I cannot solve it except for setting the ‘num workers = 0’,but my data is so big, this way is too slow. And I referred to this settings.https://github.com/Peter554/StainTools/issues/43 But it still no use. Can you help me? Thank you!
Hello,
Yes, this is unfortunately not straight forward to solve. The problem is in how CUDA is used in accelerating tasks which does not lend to multiprocessing. CUDA, does multiprocessing by splitting a task into smaller simple step tasks that can be run in parallel on GPU kernels, while CPU multiprocessing takes a tasks and runs the same task for different inputs in parallel. Unfortunately, in how CUDA is constructed these are not easily inter-mixable. This is why ‘num workers = 0’, works while setting num_workers > 0 will crash.
My code works therefore best for larger input tiles (normalizing a WSI), but not in a parallel DataLoader.
I have found that in StainTools, it is actually faster to use SPAMS settings akin to the wonderful stain_mixup project. I have used the same in my related augmentation strategy stainlist_augmentation.
Best, Cédric
