stable-diffusion-webui
stable-diffusion-webui copied to clipboard
AUTOMATIC1111
Performance improvements for NVIDIA 4x00 series
Describe what this pull request is trying to achieve. Automatically pull down new CUDA DNN binaries, and bump torch versions such that nVidia RTX4x00 series graphics cards can maximise the utilisation, as reported under issue #2449.
Additional notes and description of your changes -Bump torch version to 1.13.1+cu117 -Bump torchvision version to 0.14.1+cu117 -Use torch13 xformers ***Note: Dreambooth may be broken with this change
Don't use torch13, it breaks functionality with dream booth etc
--> only change cu113 to cu116
If U don't use dreambooth etc, this doesn't matter
_Originally posted by @sa-shiro -After torch is installed, automatically download and replace CUDA DNN binaries in torch component if NVIDIA folder is detected (i.e. assumed to be using NVIDIA graphics card).
Environment this was tested in List the environment you have developed / tested this on. As per the contributing page, changes should be able to work on Windows out of the box.
- OS: Windows
- Graphics card: NVIDIA RTX 4090
Screenshots or videos of your changes N/A
cu117 produced some odd results in training https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/5939#issuecomment-1364548943
I get issues at training hypernetworks, TI and LoRAs with torch 1.13.1+ and torchvision 0.14.1+. With 1.12.1+cu116 and 0.13.1+cu116 I don't get those issues, to take in mind.
Besides training, prompts works fine on newer torch.
Would it be possible to make this a command line argument? Something like --40xx. So that it can be easily enabled by anyone who knows they don't plan to do any training, and then disabled again when you know you do want to do any training.
good intentions, but not sure if this should be automated
a) this is windows only
b) uses hard-coded paths
c) downloading cudnn requires agreeing with nvidia's terms/conditions, doing blind download is not really best legal practice
d) cudnn install is typically a system-wide, placing it inside automatic folder is just weird
I think the best way would be to have this as an extension. TORCH_COMMAND is already what you want to set it to in the main repo and downloading/unpacking can be done in an extension installer. My main concern is something like (d), I just don't know how it will affect the users and do not want to enable it for everyone.
-Bump torch version to 1.13.1+cu117 -Bump torchvision version to 0.14.1+cu117
This is already the case.
d) cudnn install is typically a system-wide, placing it inside automatic folder is just weird
This is not a concern. Torch never uses or care if you have system-wide installation of cudnn. It always use the DLLs that come with the wheel file on its own. You can run torch + GPU without cudnn installed system-wide. The correct and only solution will be to override the DLLs that torch carries over.
a) this is windows only
This applies to Linux too, except that you need to download the linux cudnn zip and override the .so.6 files in torch folder. Maybe someone with a Linux machine can help to write this.
This applies to Linux too
not in the current form. it can be done cross-platform, but right now it isn't. but bigger issue is that i never want application to mess with my installation like that. if this is a user managed script/extension, sure, its on user. but in main repo?
mess with my installation
If you run this in a venv specifically created for webui, I see no issue, as all necessary mods will be done within this venv. If you run in a global environment and do not want such patches (and I see no reason for this since there is no harm and literally free performance), then switch to a venv or manually disable the patching code. The performance improvement it brings outweighs the potential damages (if any?) it may do to the user's system. It also greatly reduce number of unnecessary issues and discussions being raised.
There was a commit to include latest cuDNN to torch:
Add cudnn install 8.7.0.84 for CUDA 11.8 3a10bf791f53c65e4c38c29e366b45504425832a
But got reverted a few hours later:
Revert "Add cudnn install 8.7.0.84 for CUDA 11.8 62aa4e096b8414ad22796d0dbffa12988a9f3793
This applies to Linux too, except that you need to download the linux cudnn zip and override the .so.6 files in torch folder.
~I agree with @vladmandic, it makes no sense whatsoever for a third party program to overwrite system files, even less to overwrite system libraries.~
The performance improvement it brings outweighs the potential damages (if any?) it may do to the user's system.
~Imagine that gimp requires python3.9 and you have python3.10. Would it make any sense that gimp removed python3.10 or changed the symbolic link so it points to python3.9?~
~No, this would mean breaking any up to date python scripts that uses the new syntax. This scripts could be anywhere, so the performance improvement that it may bring or not, is not going to matter to a user that just had the system bricked.~
~What programs usually do is having the library included, or tell the user to update the library themselves~
It seems like you did not read my comment properly. I made it very clear that doing this is NOT system files. ALL changes are within the venv folder that you can simply rm -rf. It will NOT affect ANYTHING unless you decide to use other programs in your SD webui venv - then that's your problem.
Imagine that gimp requires python3.9 and you have python3.10. Would it make any sense that gimp removed python3.10 or changed the symbolic link so it points to python3.9?
Obviously, it makes no sense, but no one asked you to run gimp in a venv meant for SDwebui, knowing that there are SD webui specific changes. Every time you launch SD webui it will override your venv with its requirements, if not satisfied. Additionally, replacing cudnn libraries is nothing like a change from python 3.10 to 3.9. It is basically seamless and 100% compatible. No one here has reported any compatibility issues by doing this, and Nvidia specifics makes an effort to make them compatible. Both cudnn versions are compiled for 'cuda 11.x' and nvidia officially recommend always using the latest one compatible with the cuda version. So what is your point here?
Oh well, I see it now.
If you run this in a venv specifically created for webui
I assumed that you could run the webui out of venv (as you usually can on other projects) and I did not know that torch bundled the libraries. So of course, with this context now in mind, everything I said makes no sense.
Sorry for the confussion :+1:
I assumed that you could run the webui out of venv (as you usually can on other projects) and I did not know that torch bundled the libraries. So of course, with this context now in mind, everything I said makes no sense.
Yes, you can - there is nothing preventing you to run WebUI outside of venv.
And I don't use venv since i don't see a point - i have a VM for WebUI, why would I have another virtual environment inside a virtual machine.
But regardless of venv, this is a no-go unless at very least PR is updated to be cross-platform and use safe-paths.
Using hard-coded paths and being windows specific is a 110% no-go.
And what about non-CUDA environments? There is no check for that. Would you update cuDNN for AMD GPU or Intel GPU or M1 hardware?
And then there are legal implications - Torch has nVidia's permission to bundle cuDNN.
But end-user must agree to nVidia developer terms&conditions. Automatic repository has high visibility - not sure if its good idea to bypass explicitly required EULA (specifically, "NVIDIA Developer Program Membership Required").
I assumed that you could run the webui out of venv (as you usually can on other projects)
You can, but in that case it will override system wide packages, not just the cudnn we talk here but also other stuff like torch version and many dependencies. If you don't mind that, you can run it on system wide environment. In fact, I do that because I do ML stuff outside of SD webui.
Yes, you can - there is nothing preventing you to run WebUI outside of venv.
I figured, but with the given webui.sh it is by default inside of venv and my guess is that this is what its expected.
You can, but in that case it will override system wide packages, not just the cudnn
Python packages are expected to be changed, but I would not expect a python lib to not throw a version error and update a sys lib instead.