stable-diffusion-webui
stable-diffusion-webui copied to clipboard
AUTOMATIC1111
multi gpu API via ray serve
Description
-
a simple description of what you're trying to accomplish Implementing ray serve to use api across multi gpus, and with autoscaling
-
a summary of changes in code a seperate implementation of module/api/api.py thats for use with ray, its the exact same as api.py but instead of using the APIrouter() i just put each route of each function,
the modules/shared_items/shared shared class wouldnt pickleize with ray so i had to restructure and instantiate it and refactor all the uses of it to use the unstance versus the module directly
- which issues it fixes, if any
Screenshots/videos:
Checklist:
- [ x] I have read contributing wiki page
- [x ] I have performed a self-review of my own code
- [x ] My code follows the style guidelines
- [x ] My code passes tests
Well, for one, replacing all instance of shared with shared_instance is going to destroy all extensions for sure...
Supporting multiple GPUs is a possibility, but not in a way that would break extensions like this.
And adding a ray server api seems more suited for an extension.
Yea the only reason that was done is because of the way the sys.["modules.shared"].class=Shared is set, if that could be changed in a way that can be pickled then No need for that change
Hello, I am currently following this PR. Is there any progress regarding the pickled shared module?
Hello, I am currently following this PR. Is there any progress regarding the pickled shared module?
In my fork I fixed just changed up the module and seems to work fine for most part haven't been developing on it recently had other stuff pop up but am recircling on it soon as I get it deployed in Kubernetes
Hoping that this is merged soon! Would love to see it. I have an 2080 Ti and 2070S. If there's anyway in which I can help test, let me know.
jxnding: As I wrote above, support for multi-GPU would probably get merged in, but not if it's written in the way it is now, breaking all extensions. And as for ray api, I would rather it be an extension. Any additions to API that are needed to make it happen can be merged as a separate PR.
multi-gpu support for inference would be amazing
i mean like power 2 gpu combined somehow if possible like in llms for inference
new research, may helpful https://hanlab.mit.edu/blog/distrifusion https://github.com/mit-han-lab/distrifuser
