stable-diffusion-webui
stable-diffusion-webui copied to clipboard
AUTOMATIC1111
[Bug]: Addition of "cudnn.benchmark = true" breaks 2080 Ti card with CUDA Out Of Memory
Is there an existing issue for this?
- [X] I have searched the existing issues and checked the recent builds/commits
What happened?
Tried to run latest webui last night. I know it worked some weeks ago so I did a git-bisect. The PR https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/4407 which added the use of cudnn.benchmark = true, and that made my 2080 card give me CUDA out of memory errors.
Steps to reproduce the problem
- Use a 2080 Ti Card
- Try webui
- Watch CUDA memory errors
What should have happened?
It should've worked :)
And it does if I comment out the line that sets cudnn.benchmark = True
Commit where the problem happens
cd6c55c1ab14fcab15329cde599cf79e8d555657
What platforms do you use to access UI ?
Windows
What browsers do you use to access the UI ?
Google Chrome
Command Line Arguments
set COMMANDLINE_ARGS=--server-name=10.100.100.166 --theme=dark
Additional information, context and logs
I think the problem is the line:
if any([torch.cuda.get_device_capability(devid) == (7, 5) for devid in range(0, torch.cuda.device_count())]):
....not sure what it is checking (don't know enough CUDA) but it breaks my 2080 Ti card
@MasterZap I have the same card RTX 2080 Ti and I'm running the latest 685f9631b56ff8bd43bce24ff5ce0f9a0e9af490 commit but not encountering this issue. Maybe there's some way I could help diagnose.
Maybe it's CUDA version related, or maybe driver version related? I'm stuck on a slightly out-of-date driver coz my system runs Windows 8.1 ....
@MasterZap OK that seems probable. I am on Windows 10 with the most up-to-date NVIDIA drivers and Cuda version.
If it's just me, I'm fine with my local workaround. I just wonder, what the heck that line of code is supposed to do :P
This is expected. This option increases performance by reducing VRAM fragmentation at the cost of increased VRAM usage. You can just comment it out.
Could we make it an option, so I don't have to live with a one line local change?
I dont think it should be an option because you can just use the medvram or lowvram switches. The vram overhead this brings is minimal, and it is likely that you have something else also using your vram.