CUDA error: the launch timed out and was terminated

[Chen-Luan]

Describe the bug 'RuntimeError: CUDA error: the launch timed out and was terminated'

I encountered this problem when trying to generate an image of 512 * 512 and 32Sampling Steps. Later, I encountered the same problem when trying to generate an image of 512 * 512 and 4Sampling Steps (CFG Scale is 20, and other parameters are the default values)

...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [01:22<00:00, 20.62s/it]
Error completing request█████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [01:01<00:00, 16.68s/it]
Arguments: ('white hear', 'lowres,bad anatomy,bad hands,text,error,missing fngers,extra digt,fewer digits,cropped,wort quality,low quality,normal quality, jpeg artifacts,signature,watermark,username,blurry,bad feet', 'None', 'None', 4, 0, False, False, 1, 1, 20, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, False, 0.7, 0, False, False, None, '', 1, '', 4, '', True, False) {}
Traceback (most recent call last):
  File "E:\GitHubDesktop\stable-diffusion-webui\modules\ui.py", line 176, in f
    res = list(func(*args, **kwargs))
  File "E:\GitHubDesktop\stable-diffusion-webui\webui.py", line 68, in f
    res = func(*args, **kwargs)
  File "E:\GitHubDesktop\stable-diffusion-webui\modules\txt2img.py", line 43, in txt2img
    processed = process_images(p)
  File "E:\GitHubDesktop\stable-diffusion-webui\modules\processing.py", line 409, in process_images
    lowvram.send_everything_to_cpu()
  File "E:\GitHubDesktop\stable-diffusion-webui\modules\lowvram.py", line 13, in send_everything_to_cpu
    module_in_gpu.to(cpu)
  File "E:\GitHubDesktop\stable-diffusion-webui\venv\lib\site-packages\pytorch_lightning\core\mixins\device_dtype_mixin.py", line 113, in to
    return super().to(*args, **kwargs)
  File "E:\GitHubDesktop\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 927, in to
    return self._apply(convert)
  File "E:\GitHubDesktop\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
    module._apply(fn)
  File "E:\GitHubDesktop\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
    module._apply(fn)
  File "E:\GitHubDesktop\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 602, in _apply
    param_applied = fn(param)
  File "E:\GitHubDesktop\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 925, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

venv "E:\GitHubDesktop\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.7 (tags/v3.10.7:6cc6b13, Sep  5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)]
Commit hash: 45fbd1c5fec887988ab555aac75a999d4f3aff40
Traceback (most recent call last):
  File "E:\GitHubDesktop\stable-diffusion-webui\launch.py", line 164, in <module>
    prepare_enviroment()
  File "E:\GitHubDesktop\stable-diffusion-webui\launch.py", line 120, in prepare_enviroment
    run_python("import torch; assert torch.cuda.is_available(), 'Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check'")
  File "E:\GitHubDesktop\stable-diffusion-webui\launch.py", line 56, in run_python
    return run(f'"{python}" -c "{code}"', desc, errdesc)
  File "E:\GitHubDesktop\stable-diffusion-webui\launch.py", line 32, in run
    raise RuntimeError(message)
RuntimeError: Error running command.
Command: "E:\GitHubDesktop\stable-diffusion-webui\venv\Scripts\python.exe" -c "import torch; assert torch.cuda.is_available(), 'Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check'"
Error code: 1
stdout: <empty>
stderr: Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check

Desktop (please complete the following information):

• OS: Windows11
• Browser: chrome
• Commit revision 45fbd1c5fec887988ab555aac75a999d4f3aff40

Additional context GPU 0: ntel(R) HD Graphics 620 GPU 1: NVIDIA GeForce 940MX (2GB) parameter: --lowvram --always-batch-cond-uncond --opt-split-attention

[Chen-Luan]

Possible related problems #2085

[9CB797FF-9380-45F2-BB88-BB86CA0E32BF]

In my case, this solved it. COMMANDLINE_ARGS= --lowvram --always-batch-cond-uncond --precision full --no-half to COMMANDLINE_ARGS= --lowvram

[HydrogenRb]

for me I have :

Preparing dataset... 0% 0/128 [00:00<?, ?it/s] Applying cross attention optimization (Doggettx). Error completing request Arguments: ('SyntheticDawn', '0.005', '/content/drive/MyDrive/ai_test/SyntheticDawn/Oct_13_train_test_tag_2', 'textual_inversion', 512, 512, 1000, 500, 500, '/content/stable-diffusion-webui/textual_inversion_templates/style_filewords.txt', True, '') {} Traceback (most recent call last): File "/content/stable-diffusion-webui/modules/ui.py", line 186, in f res = list(func(*args, **kwargs)) File "/content/stable-diffusion-webui/webui.py", line 64, in f res = func(*args, **kwargs) File "/content/stable-diffusion-webui/modules/textual_inversion/ui.py", line 31, in train_embedding embedding, filename = modules.textual_inversion.textual_inversion.train_embedding(*args) File "/content/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 207, in train_embedding ds = modules.textual_inversion.dataset.PersonalizedBase(data_root=data_root, width=training_width, height=training_height, repeats=shared.opts.training_image_repeats_per_epoch, placeholder_token=embedding_name, model=shared.sd_model, device=devices.device, template_file=template_file) File "/content/stable-diffusion-webui/modules/textual_inversion/dataset.py", line 71, in init torchdata = torch.from_numpy(npimage).to(device=device, dtype=torch.float32) RuntimeError: CUDA error: unspecified launch failure CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

============================ I change the version of CUDA to solve it.

[DoomKrakken]

I change the version of CUDA to solve it

What version of CUDA are you using?

In my case, this solved it. COMMANDLINE_ARGS= --lowvram --always-batch-cond-uncond --precision full --no-half to COMMANDLINE_ARGS= --lowvram

Where did you apply this?

[Chen-Luan]

In my case, this solved it. COMMANDLINE_ARGS= --lowvram --always-batch-cond-uncond --precision full --no-half to COMMANDLINE_ARGS= --lowvram

I tried, but it didn't work

[judgeou]

Ubuntu, same problem

[judgeou]

Hope #3494 can solve the problem

relate 72383abacdc6a101704a6f73758ce4d0bb68c9d1, e38625011cd4955da4bc67fe95d1d0f4c0c53899

[ngoclong19]

To solve this issue, I set the environment variable CUDA_LAUNCH_BLOCKING=1 before running the script.

It doesn't seem to cause any performance drop on inference.

[Xeltosh]

Where do i input this variable? Have the same problem and i am rather new to this.

[ngoclong19]

Where do i input this variable?

For Windows, in the file webui-user.bat, add this line set CUDA_LAUNCH_BLOCKING=1 after the line set COMMANDLINE_ARGS=.

If you are using Linux, in the file webui-user.sh, add this line export CUDA_LAUNCH_BLOCKING=1 at the end.

[okin89]

Where do i input this variable?

For Windows, in the file webui-user.bat, add this line set CUDA_LAUNCH_BLOCKING=1 after the line set COMMANDLINE_ARGS=.

If you are using Linux, in the file webui-user.sh, add this line export CUDA_LAUNCH_BLOCKING=1 at the end.

Mind telling me where exactly i can find this file for windows? I tried searching for it into the meshroom files but i can't find it.

[ngoclong19]

@okin89, the file is at https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/webui-user.bat

I don't know about Meshroom. We use the batch file approach so it only affect this tool. Maybe you could follow this guide to set the variable CUDA_LAUNCH_BLOCKING system wide: https://docs.oracle.com/en/database/oracle/machine-learning/oml4r/1.5.1/oread/creating-and-modifying-environment-variables-on-windows.html

[okin89]

@okin89, the file is at https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/webui-user.bat

I don't know about Meshroom. We use the batch file approach so it only affect this tool. Maybe you could follow this guide to set the variable CUDA_LAUNCH_BLOCKING system wide: https://docs.oracle.com/en/database/oracle/machine-learning/oml4r/1.5.1/oread/creating-and-modifying-environment-variables-on-windows.html

Thank you for helping me, i didn't noticed this is a different program, i was reading this topic and lead me here

https://github.com/alicevision/Meshroom/issues/1997