waifu-diffusion icon indicating copy to clipboard operation
waifu-diffusion copied to clipboard

Published 20 hours ago • verified publisher icon harubaru

[FEATURE REQUEST] Add multi-threading/processing to the Dataset Validator

Open chavinlo opened this issue 2 years ago • 5 comments

It's awfully slow.

I am willing to do it if someone gives me directions

chavinlo avatar Nov 29 '22 04:11 chavinlo

Did you install xformers? https://github.com/huggingface/diffusers/issues/1343

chinoll avatar Nov 30 '22 12:11 chinoll

Did you install xformers? huggingface/diffusers#1343

yes but I don't think xformers has anything to do with the validation process...

chavinlo avatar Dec 01 '22 00:12 chavinlo

You can skip validation, which is what I would suggest. Internally, I'm personally using a preprocessing application written in Rust. This probably isn't something you want to do in Python.

cafeai avatar Dec 01 '22 07:12 cafeai

small self plug for a preprocessor written in python and fully parallel: https://github.com/lopho/parallel_dataprocessor

lopho avatar Dec 05 '22 03:12 lopho

#60 has parallel validation and migration

lopho avatar Dec 05 '22 09:12 lopho