stable-diffusion-webui
stable-diffusion-webui copied to clipboard
AUTOMATIC1111
[Bug]: Slowdown over time in v1.6.0
Is there an existing issue for this?
- [X] I have searched the existing issues and checked the recent builds/commits
What happened?
I don't know how to exactly explain this, but a friend and I have been using 1.6.0 for a couple of weeks now, enjoying the amazing speed increase compared to the previous installments (my speed average went from 7 minutes 28 seconds per image to 3 minutes 48 seconds as an example), but then slowly after a week of use, without changing anything, both of us started to notice a sudden increase in time needed to generate an Image.. These images are from my experience
After a week of using the install
After a week and half of using the install
Slow but gradual increase over time, we figured something was off with the install and just reverted back to the previous checkpoint we were using before moving to 1.6.0.
After 4-5 days, my friend returned to 1.6.0 and noticed the speeds went back to normal like when he first updated to 1.6.0 for the first time, so we deduced there is something in 1.6.0 that causes the generating speed to slowly but gradually increase the longer you use it. This happens to both txt2img and img2img separately as well, which adds to the confusion.
The hardest part is figuring what would be the cause of this weird phenomenon?
Steps to reproduce the problem
- Update to v1.6.0
- Open webui-user.bat
- Generate images
- Close webui-user.bat
- Repeat steps 2-4 over the span of a week or two.
- ???
What should have happened?
Generating speed shouldn't be slowing down gradually day by day, It would be logical if it was a hardware issue or something but reverting and updating shouldn't bring back the original speed then.
Sysinfo
What browsers do you use to access the UI ?
Google Chrome
Console logs
venv "G:\Model\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.6.0
Commit hash: 5ef669de080814067961f28357256e8fe27544f4
Launching Web UI with arguments: --enable-insecure-extension-access --autolaunch
no module 'xformers'. Processing without...
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
Loading weights [63da30d731] from G:\Model\stable-diffusion-webui\models\Stable-diffusion\MadnessMix-Pixiv.safetensors
Creating model from config: G:\Model\stable-diffusion-webui\configs\v1-inference.yaml
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Startup time: 16.5s (prepare environment: 3.2s, import torch: 5.3s, import gradio: 1.3s, setup paths: 1.2s, initialize shared: 0.3s, other imports: 0.9s, setup codeformer: 0.2s, load scripts: 2.7s, create ui: 0.6s, gradio launch: 0.6s).
Loading VAE weights specified in settings: G:\Model\stable-diffusion-webui\models\VAE\kl-f8-anime2.vae.pt
Applying attention optimization: Doggettx... done.
Model loaded in 5.9s (load weights from disk: 0.8s, create model: 0.6s, apply weights to model: 2.7s, apply half(): 1.2s, load VAE: 0.3s, calculate empty prompt: 0.3s).
X/Y/Z plot will create 4 images on 1 4x1 grid. (Total steps to process: 240)
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:26<00:00, 1.12it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [03:56<00:00, 7.90s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:24<00:00, 1.23it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [04:07<00:00, 8.24s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:23<00:00, 1.25it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [04:06<00:00, 8.21s/it]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:24<00:00, 1.25it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [03:37<00:00, 7.26s/it]
Total progress: 100%|████████████████████████████████████████████████████████████████| 240/240 [18:23<00:00, 4.60s/it]
Total progress: 100%|████████████████████████████████████████████████████████████████| 240/240 [18:23<00:00, 7.26s/it]
Additional information
Note that the console logs is after reverting back to 1.6.0 myself, so the screenshots provided earlier wont match to the gen times in them. I have tried deleting stuff like the venv folder and updating extensions as well if that matters.
I'm using Windows10 and Google-Chrome and is not seeing any slow-down here.
But I have noticed people talking about the way Windows behave when "Windows Fast Startup" is ON, in that it technically works like Windows is never actually re-started.
Only if a user manually selects "RESTART" the PC will reset its up-time and re-load drivers and system from scratch.
The normal shut-down and start-up method people use, maintains the PC in a kind of limbo where the system is not actually fully reset at start-up.
Perhaps the effect you see over time is the result of this?
If you don't have "Windows Fast Startup" active on your PC (And it is ON by default, so unless you manually switch if OFF it is active) then this of course does not apply to you.
But otherwise; try an actual RESTART of your PC (Not just a normal shut-down and start-up, but a RE-start)
Maybe your issue is something else, but just thought it was worth mentioning this :)
(You can check your PC's 'up-time' by opening "Task Manager" and go to the tab "Performance" and click the CPU-tab. If you have "Windows Fast Startup" active, and only use the normal shut-down/start-up method to power down and power up your PC, your PC may state its up-time as being not just hours but days. Only an actual RE-start will reset this and flush the caches and re-load the system)
Same Issue here if i use upscaller before it was
4 512x512 to 1024x1024 takes 1.5 min now it takes 6 min
i will try revert to version before this on my end to see if the issue is not there.
found the fix can you try checking this
make sure the refiner is set to 1 on SD 1.5
I think we should have one box to force turn on/off all SDXL options so this will not be affected
and any changes to code should be on separate processes if it will cause issues to SD 1.5
Do you have your vram usage partially in shared memory after large generations?
found the fix can you try checking this make sure the refiner is set to 1 on SD 1.5 I think we should have one box to force turn on/off all SDXL options so this will not be affected and any changes to code should be on separate processes if it will cause issues to SD 1.5
Weirdly enough, testing this caused it to revert back to the slower times after my revert-to-update fix that my friend notified me about.. straight went from 3min 43 seconds-4min ~seconds to this
Do you have your vram usage partially in shared memory after large generations?
What exactly do you mean by this? like was this an option I might have turned on or something? Sorry about my cluelessness, its probably just me being half asleep that I'm scratching my head with this question..
Dunno why but my it/s went down from 9.17 it/s to 7.43 it/s. It's a noticeable difference, when using hires fix it's like 30 seconds... For no reason
Started from 11s, went up to 50s. After a pause it came down to 18s. After another pause it came down to 13s.
Red lines are where I stopped generating for a while.
It turns out it was GPU temperature causing the slow down. This makes sense too because when I pause for a while, GPU temp goes down causing the generation time to go up again.
For me GPU throttling starts from around ~80C. I recommend opening Task Manager > Performance tab and keep an eye on temperature (or use some other tool).
Do you have your vram usage partially in shared memory after large generations?
What exactly do you mean by this? like was this an option I might have turned on or something? Sorry about my cluelessness, its probably just me being half asleep that I'm scratching my head with this question..
It finally clicked for me, my larger gens vary between 4-7GB of vRAM usage, barely below the maximum as my card has (8GB) so it shouldn't trigger shared memory.. doesn't really explain why my friend has the same issue (hes running a 24GB card) and doesn't explain why reverting to an older version and going back without changing anything reverts the speeds back to normal. Though I did go check my driver number and I am on 536.23 so there is a possibility something about it still lingers and affects the gradual slowdown.
It turns out it was GPU temperature causing the slow down. This makes sense too because when I pause for a while, GPU temp goes down causing the generation time to go up again.
For me GPU throttling starts from around ~80C. I recommend opening Task Manager > Performance tab and keep an eye on temperature (or use some other tool).
The issue me and a few others (seen more people report this exact interaction after switching to 1.6.0) are having sadly isn't based on this issue, but I am glad you figured out what was causing the problem for yours :)
Только реальный повторный запуск сбросит это значение, очистит кеши и перезагрузит систему
It`s help me. Thank you
