AUTOMATIC1111
Huge performance decrease with latest Nvidia drivers
Describe the bug Since updating to the latest Nvidia drivers (522.25), time to generate images has increased massively. It's gone from around 4it/s to below 1it/s and doesn't appear to be using my GPU fully like it used to. I can only describe in laypersons terms, but I've been running with an RTX 3080 12GB and before the driver update it used to be visibly working harder, would reach higher temps, fans would have to go faster etc, but now it never goes higher than it's idling temp and performance manager shows only around 20% utilisation rather than the 70-80% it would reach previously. There are no error messages or anything so I can't give more information beyond "it's going a lot slower" unfortunately. Having done a little investigation and checking MSI Afterburner it seems my GPU usage is hitting 100%, but the core clock isn't budging above 240MHz. It also appears to be capping at around 9GB of VRAM, whereas before it had no problem utilising up to 11.5GB; performance manager shows nothing else is using my VRAM right now either, so the extra ~3GB is going unused.
EDIT: After rolling back to a previous commit and finding no change, I tried rolling back my nvidia driver from 522.25 to 517.48 and everything worked fine again. It seems the issue is definitely with the latest nvidia driver.
To Reproduce Steps to reproduce the behavior:
- Update to latest Nvidia driver (522.25)
- Generate any image on any setting
Expected behavior Generates images at the speed it was going previously (around 4it/s)
Desktop:
- OS: Windows 10
- Browser: Chrome
- Commit revision: bb0e7232b301d1706bbd0e09367dece3bb7ac07c
- GPU: RTX 3080 12GB, Nvidia driver 522.25
Seems like it is "stuck" at P8 state, the low power state. P0 would be the highest. And P2 seems to be used when on CUDA workloads. Did you try to change in nvidia settings the Power management mode to High performance? I also noticed this and thought Nvidia driver is acting weird when switching between power states. I had this setting set to Normal, and then I switched to High Performance. I think this solved my issue but not sure if this is the solution.
To check the P-state your GPU is in, you can execute this in a terminal:
nvidia-smi -q -d PERFORMANCE
If Performance State is in P2 or P0, then it is good.
Also this should tell you why the low clock in the Clocks Throttle Reasons.
In case it doesn't work, maybe this command can be useful (although this seems more for memory clock cap in CUDA workloads, I don't think this is the problem in your case): https://forums.developer.nvidia.com/t/titan-v-slower-than-1080ti-tensorflow-18-08-py3-and-396-54-drivers/65448/13?u=5wpa2kkd
Huh, it seems like the act of rolling back to 517.48 and then updating back up to 522.25 has fixed the issue. Running it on 522.25 now has it working at the same speed it was before this issue started, and is fully utilising my GPU again. I ran the command you gave before starting the UI, after starting it, and during generation and it all seems okay for now. Without UI running: P0 With UI running: P2 During generation: P2
Clock speeds back up to their normal 1950MHz during generation, too. I'm not sure what the problem is, but it appears rolling back and then reupdating my Nvidia driver has fixed it, at least for now. I'll try restarting my PC and see if it comes back at all.
Okay yeah, I restarted my PC and everything is still running fine. I have no idea what the problem was, only that it was caused by updating to 522.25, and that it seems to have been fixed by rolling back to a previous version and then updating back up.
If anything this actually reminded me to set my PC back to "High Performance" which actually made everything faster with SD. I turned it down for the summer heatwave. So thanks for that, also installed the new drivers and everything is working well.
I have also set my PC to "High Performance" mode, wow. Hold on Chewbacca, light speed on ;)
