[Bug]: generation process hangs randomly

[janwilmans]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

When running "generate forever" the process stops mid-generation of an image. The screenshot shows 50%, but I have also seen it at 20% for example.

This started happening after updating from 22bcc7be428c94e9408f589966c2040187245d81 to 5ab7f213bec2f816f9c5644becb32eb72c8ffb89

which included a new xformers and torch version

Steps to reproduce the problem

1. Generate forever

What should have happened?

keep generating

Commit where the problem happens

5ab7f213bec2f816f9c5644becb32eb72c8ffb89

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--api --xformers --autostart

List of extensions

na

Console logs

Successfully installed fastapi-0.90.1 starlette-0.23.1

C:\ai\stable-diffusion-webui>webui-user.bat
venv "C:\ai\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Commit hash: 5ab7f213bec2f816f9c5644becb32eb72c8ffb89
Installing requirements
Installing sd-dynamic-prompts requirements.txt


Launching Web UI with arguments: --xformers --api --autolaunch
Additional Network extension not installed, Only hijack built-in lora
LoCon Extension hijack built-in lora successfully
Image Browser: ImageReward is not installed, cannot be used.
Loading weights [0466ae65a0] from C:\ai\stable-diffusion-webui\models\Stable-diffusion\a\epicDiffusion11.safetensors
Creating model from config: C:\ai\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(14): bad-artist-anime, bad-hands-5, bad_prompt, bad_prompt_version2, bukkakAI, corneo_cowgirl, corneo_pov_anal, easynegative, ng_deepnegative_v1_75t, realcumAI, reverseSuspended_analVersion, sofiavdv, Style-Goblinmode, style-nebmagic
Textual inversion embeddings skipped(4): CGI_Animation, midjourney, remix, VikingPunk
Model loaded in 3.1s (load weights from disk: 0.2s, create model: 0.3s, apply weights to model: 0.6s, apply half(): 0.6s, move model to device: 0.5s, load textual inversion embeddings: 0.9s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 11.5s (import torch: 1.5s, import gradio: 0.9s, import ldm: 0.5s, other imports: 0.8s, list SD models: 0.2s, load scripts: 1.1s, load SD checkpoint: 3.3s, create ui: 2.9s, gradio launch: 0.2s, scripts app_started_callback: 0.1s).
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:03<00:00,  3.33it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  5.66it/s]
{"prompt": "superman ring", "all_prompts": ["superman ring"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 1583041484, "all_seeds": [1583041484], "subseed": 2282960185, "all_subseeds": [2282960185], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "Euler a", "cfg_scale": 7, "steps": 10, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_hash": "0466ae65a0", "seed_resize_from_w": 0, "seed_resize_from_h": 0, "denoising_strength": null, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["superman ring\nSteps: 10, Sampler: Euler a, CFG scale: 7, Seed: 1583041484, Size: 512x512, Model hash: 0466ae65a0, Model: a_epicDiffusion11"], "styles": [], "job_timestamp": "20230507203841", "clip_skip": 1, "is_using_inpainting_conditioning": false}
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  6.21it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  5.83it/s]
{"prompt": "superman ring", "all_prompts": ["superman ring"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 3570989737, "all_seeds": [3570989737], "subseed": 2608206027, "all_subseeds": [2608206027], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "Euler a", "cfg_scale": 7, "steps": 10, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_hash": "0466ae65a0", "seed_resize_from_w": 0, "seed_resize_from_h": 0, "denoising_strength": null, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["superman ring\nSteps: 10, Sampler: Euler a, CFG scale: 7, Seed: 3570989737, Size: 512x512, Model hash: 0466ae65a0, Model: a_epicDiffusion11"], "styles": [], "job_timestamp": "20230507203852", "clip_skip": 1, "is_using_inpainting_conditioning": false}
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  6.13it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  5.64it/s]
{"prompt": "superman ring", "all_prompts": ["superman ring"], "negative_prompt": "", "all_negative_prompts": [""], "seed": 1517543420, "all_seeds": [1517543420], "subseed": 2692408255, "all_subseeds": [2692408255], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "Euler a", "cfg_scale": 7, "steps": 10, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_hash": "0466ae65a0", "seed_resize_from_w": 0, "seed_resize_from_h": 0, "denoising_strength": null, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": ["superman ring\nSteps: 10, Sampler: Euler a, CFG scale: 7, Seed: 1517543420, Size: 512x512, Model hash: 0466ae65a0, Model: a_epicDiffusion11"], "styles": [], "job_timestamp": "20230507203903", "clip_skip": 1, 

---snip--

{"prompt": "superman ring", "all_prompts": ["superman ring"], "negative_prompt": "bracelet, chain", "all_negative_prompts": ["bracelet, chain"], "seed": 2367207477, "all_seeds": [2367207477], "subseed": -1, "all_subseeds": [1873248564], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "Euler a", "cfg_scale": 7, "steps": 10, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_hash": "1a189f0be6", "seed_resize_from_w": 0, "seed_resize_from_h": 0, "denoising_strength": null, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": [""], "styles": [], "job_timestamp": "20230507210432", "clip_skip": 1, "is_using_inpainting_conditioning": false}
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00,  2.16it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00,  2.22it/s]
{"prompt": "superman ring", "all_prompts": ["superman ring"], "negative_prompt": "bracelet, chain", "all_negative_prompts": ["bracelet, chain"], "seed": 113154756, "all_seeds": [113154756], "subseed": -1, "all_subseeds": [3758114495], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "Euler a", "cfg_scale": 7, "steps": 10, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_hash": "1a189f0be6", "seed_resize_from_w": 0, "seed_resize_from_h": 0, "denoising_strength": null, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": [""], "styles": [], "job_timestamp": "20230507210437", "clip_skip": 1, "is_using_inpainting_conditioning": false}
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00,  2.20it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00,  2.29it/s]
{"prompt": "superman ring", "all_prompts": ["superman ring"], "negative_prompt": "bracelet, chain", "all_negative_prompts": ["bracelet, chain"], "seed": 3584773216, "all_seeds": [3584773216], "subseed": -1, "all_subseeds": [2398568627], "subseed_strength": 0, "width": 512, "height": 512, "sampler_name": "Euler a", "cfg_scale": 7, "steps": 10, "batch_size": 1, "restore_faces": false, "face_restoration_model": null, "sd_model_hash": "1a189f0be6", "seed_resize_from_w": 0, "seed_resize_from_h": 0, "denoising_strength": null, "extra_generation_params": {}, "index_of_first_image": 0, "infotexts": [""], "styles": [], "job_timestamp": "20230507210443", "clip_skip": 1, "is_using_inpainting_conditioning": false}
 50%|█████████████████████████████████████████▌                                         | 5/10 [00:02<00:02,  2.34it/s]
Total progress:  50%|█████████████████████████████████▌                                 | 5/10 [00:01<00:01,  2.68it/s] 

### Additional information

I should mention that I was seeing #7759 so I downgraded to `fastapi-0.90.1` to work around that

[janwilmans]

temporarily worked around it by downgrading:

git checkout 22bcc7be428c94e9408f589966c2040187245d81
venv\Scripts\pip3 --require-virtualenv uninstall --yes torch
venv\Scripts\pip3 --require-virtualenv uninstall --yes clean-fid
venv\Scripts\pip3 --require-virtualenv uninstall --yes xformers
venv\Scripts\pip3 --require-virtualenv uninstall --yes torchvision

and then Launching Web UI with arguments: --reinstall-torch

[halr9000]

Looks like dupe of #10110 .

[oliverban]

This happens here to. I'm also on torch 2.0. Randomly hangs during generation and the GPU spikes to 100% and stays there without any fluctuations.

[marcsyp]

I also have this problem and reverting to the master branch deployment at the above commit does indeed solve the problems -- but of course this is far less than ideal as a solution, as there has been a lot of development in the last 5 weeks and we are out in the wind...

[nickr61]

same issue after updating to torch 2, it seems to hang on simple prompts for me, more complex ones run on in generate forever ok but if i use a very few words it hangs after a few image generations and I have to close the cmd window and restart with webui-user.bat, just reloading the web ui doesn't work

[halr9000]

@janwilmans could you please close this issue as a duplicate of #10110 so that people won't continue to comment on this one and miss any updates?

[janwilmans]

duplicate of #10110