[Bug]: CUDA error: the launch timed out and was terminated

[ZeroCool22]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

Traceback (most recent call last): File "C:\Users\ZeroCool22\Desktop\Auto\modules\call_queue.py", line 45, in f res = list(func(*args, **kwargs)) File "C:\Users\ZeroCool22\Desktop\Auto\modules\call_queue.py", line 25, in f shared.state.begin() File "C:\Users\ZeroCool22\Desktop\Auto\modules\shared.py", line 193, in begin devices.torch_gc() File "C:\Users\ZeroCool22\Desktop\Auto\modules\devices.py", line 59, in torch_gc torch.cuda.empty_cache() File "C:\Users\ZeroCool22\Desktop\Auto\venv\lib\site-packages\torch\cuda\memory.py", line 121, in empty_cache torch._C._cuda_emptyCache() RuntimeError: CUDA error: the launch timed out and was terminated CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Steps to reproduce the problem

1. Generate images for a long period of time.

What should have happened?

Keep working.

Commit where the problem happens

685f9631b56ff8bd43bce24ff5ce0f9a0e9af490

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

set COMMANDLINE_ARGS=--xformers --vae-path "C:\Users\ZeroCool22\Desktop\Auto\models\Stable-diffusion\vae-ft-mse-840000-ema-pruned.ckpt"

Additional information, context and logs

GPU: 1080 TI

[TheMundaneDave]

I'm not sure but this looks suspiciously like the same problem I've had for weeks now. Reported in #5097

[NaughtDZ]

Allow me ask one thing:Are you install sd_dreambooth_extension? I was find a solution: Clone a new webui project. But Don't start it . First, add the following in web-ui.bat: set CUDA_DEVICE_ORDER=PCI_BUS_ID set CUDA_VISIBLE_DEVICES=0 (you can type "nvidia-smi" in windows cmd to get what you wanna use GPU devices number,for me,M40 is 0 ,3070 is 1) Save and run web-ui.bat

But when I install sd_dreambooth_extension.M40 cant work again.