ComfyUI
ComfyUI copied to clipboard
Published
5 days ago
•
comfyanonymous
comfyanonymous
Segmentation fault (core dumped)
Environment
OS: Ubuntu 22.04.2 LTS (Jammy Jellyfish)
GPU: AMD Radeon RX 6600
Python: 3.10.12
Models used: sd_xl_base_1.0.safetensors, sd_xl_refiner_1.0.safetensors and v1-5-pruned.safetensors
Libraries (from pip install -r requirements.txt):
torch (2.0.1+rocm5.4.2)
torchsde(0.2.5)
einops (0.6.1)
transformers>=4.25.1 (4.31.0)
safetensors>=0.3.0 (0.3.1)
aiohttp (3.8.5)
accelerate (0.21.0)
pyyaml (5.4.1)
Pillow (9.0.1)
scipy (1.11.1)
tqdm (4.65.0)
jinja2 (3.1.2)
typing-extensions (4.7.1)
networkx (3.1)
filelock (3.12.2)
pytorch-triton-rocm<2.1,>=2.0.0 (2.0.1)
sympy (1.12)
boltons>=20.2.1 (23.0.0)
trampoline>=0.1.2 (0.1.2)
numpy>=1.19.* (1.24.1)
requests (2.25.1)
tokenizers!=0.11.3,<0.14,>=0.11.1 (0.13.3)
regex!=2019.12.17 (2023.6.3)
huggingface-hub<1.0,>=0.14.1 (0.16.4)
packaging>=20.0 (23.1)
attrs>=17.3.0 (23.1.0)
frozenlist>=1.1.1 (1.4.0)
aiosignal>=1.1.2 (1.3.1)
charset-normalizer<4.0,>=2.0 (3.2.0)
async-timeout<5.0,>=4.0.0a3 (4.0.2)
multidict<7.0,>=4.5 (6.0.4)
yarl<2.0,>=1.0 (1.9.2)
psutil (5.9.5)
fsspec (2023.6.0)
cmake (3.26.4)
lit (16.0.6)
idna>=2.0 (3.3)
MarkupSafe>=2.0n(2.0.1)
mpmath>=0.19 (1.3.0)
Error description
After executing python3 main.py, it will start thinking for a while and then throw a dump core. It goes through the "Load Checkpoint" job without errors. Then it fails shortly after entering "CLIP Text Encode (prompt)".
This is the workflow:
This is the information displayed when using SDXL models:
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon RX 6600 : native
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
Starting server
To see the GUI go to: http://127.0.0.1:8188
got prompt
model_type EPS
adm 2816
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
Segmentation fault (core dumped)
When using SD1.5 the information is the following:
Total VRAM 8176 MB, total RAM 15923 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon RX 6600 : native
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
Starting server
To see the GUI go to: http://127.0.0.1:8188
got prompt
model_type EPS
adm 0
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
left over keys: dict_keys(['alphas_cumprod', 'alphas_cumprod_prev', 'betas', 'cond_stage_model.transformer.text_model.embeddings.position_ids', 'log_one_minus_alphas_cumprod', 'model_ema.decay', 'model_ema.diffusion_modelinput_blocks00bias', 'model_ema.diffusion_modelinput_blocks00weight', 'model_ema.diffusion_modelinput_blocks100emb_layers1bias', 'model_ema.diffusion_modelinput_blocks100emb_layers1weight', 'model_ema.diffusion_modelinput_blocks100in_layers0bias', 'model_ema.diffusion_modelinput_blocks100in_layers0weight', 'model_ema.diffusion_modelinput_blocks100in_layers2bias', 'model_ema.diffusion_modelinput_blocks100in_layers2weight', 'model_ema.diffusion_modelinput_blocks100out_layers0bias', 'model_ema.diffusion_modelinput_blocks100out_layers0weight', 'model_ema.diffusion_modelinput_blocks100out_layers3bias', 'model_ema.diffusion_modelinput_blocks100out_layers3weight', 'model_ema.diffusion_modelinput_blocks10emb_layers1bias', 'model_ema.diffusion_modelinput_blocks10emb_layers1weight', 'model_ema.diffusion_modelinput_blocks10in_layers0bias', 'model_ema.diffusion_modelinput_blocks10in_layers0weight', 'model_ema.diffusion_modelinput_blocks10in_layers2bias', 'model_ema.diffusion_modelinput_blocks10in_layers2weight', 'model_ema.diffusion_modelinput_blocks10out_layers0bias', 'model_ema.diffusion_modelinput_blocks10out_layers0weight', 'model_ema.diffusion_modelinput_blocks10out_layers3bias', 'model_ema.diffusion_modelinput_blocks10out_layers3weight', 'model_ema.diffusion_modelinput_blocks110emb_layers1bias', 'model_ema.diffusion_modelinput_blocks110emb_layers1weight', 'model_ema.diffusion_modelinput_blocks110in_layers0bias', 'model_ema.diffusion_modelinput_blocks110in_layers0weight', 'model_ema.diffusion_modelinput_blocks110in_layers2bias', 'model_ema.diffusion_modelinput_blocks110in_layers2weight', 'model_ema.diffusion_modelinput_blocks110out_layers0bias', 'model_ema.diffusion_modelinput_blocks110out_layers0weight', 'model_ema.diffusion_modelinput_blocks110out_layers3bias', 'model_ema.diffusion_modelinput_blocks110out_layers3weight', 'model_ema.diffusion_modelinput_blocks11normbias', 'model_ema.diffusion_modelinput_blocks11normweight', 'model_ema.diffusion_modelinput_blocks11proj_inbias', 'model_ema.diffusion_modelinput_blocks11proj_inweight', 'model_ema.diffusion_modelinput_blocks11proj_outbias', 'model_ema.diffusion_modelinput_blocks11proj_outweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0ffnet2bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0ffnet2weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm1bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm1weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm2bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm2weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm3bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm3weight', 'model_ema.diffusion_modelinput_blocks20emb_layers1bias', 'model_ema.diffusion_modelinput_blocks20emb_layers1weight', 'model_ema.diffusion_modelinput_blocks20in_layers0bias', 'model_ema.diffusion_modelinput_blocks20in_layers0weight', 'model_ema.diffusion_modelinput_blocks20in_layers2bias', 'model_ema.diffusion_modelinput_blocks20in_layers2weight', 'model_ema.diffusion_modelinput_blocks20out_layers0bias', 'model_ema.diffusion_modelinput_blocks20out_layers0weight', 'model_ema.diffusion_modelinput_blocks20out_layers3bias', 'model_ema.diffusion_modelinput_blocks20out_layers3weight', 'model_ema.diffusion_modelinput_blocks21normbias', 'model_ema.diffusion_modelinput_blocks21normweight', 'model_ema.diffusion_modelinput_blocks21proj_inbias', 'model_ema.diffusion_modelinput_blocks21proj_inweight', 'model_ema.diffusion_modelinput_blocks21proj_outbias', 'model_ema.diffusion_modelinput_blocks21proj_outweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0ffnet2bias', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0ffnet2weight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0norm1bias', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0norm1weight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0norm2bias', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0norm2weight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0norm3bias', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0norm3weight', 'model_ema.diffusion_modelinput_blocks30opbias', 'model_ema.diffusion_modelinput_blocks30opweight', 'model_ema.diffusion_modelinput_blocks40emb_layers1bias', 'model_ema.diffusion_modelinput_blocks40emb_layers1weight', 'model_ema.diffusion_modelinput_blocks40in_layers0bias', 'model_ema.diffusion_modelinput_blocks40in_layers0weight', 'model_ema.diffusion_modelinput_blocks40in_layers2bias', 'model_ema.diffusion_modelinput_blocks40in_layers2weight', 'model_ema.diffusion_modelinput_blocks40out_layers0bias', 'model_ema.diffusion_modelinput_blocks40out_layers0weight', 'model_ema.diffusion_modelinput_blocks40out_layers3bias', 'model_ema.diffusion_modelinput_blocks40out_layers3weight', 'model_ema.diffusion_modelinput_blocks40skip_connectionbias', 'model_ema.diffusion_modelinput_blocks40skip_connectionweight', 'model_ema.diffusion_modelinput_blocks41normbias', 'model_ema.diffusion_modelinput_blocks41normweight', 'model_ema.diffusion_modelinput_blocks41proj_inbias', 'model_ema.diffusion_modelinput_blocks41proj_inweight', 'model_ema.diffusion_modelinput_blocks41proj_outbias', 'model_ema.diffusion_modelinput_blocks41proj_outweight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0ffnet2bias', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0ffnet2weight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0norm1bias', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0norm1weight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0norm2bias', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0norm2weight', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0norm3bias', 'model_ema.diffusion_modelinput_blocks41transformer_blocks0norm3weight', 'model_ema.diffusion_modelinput_blocks50emb_layers1bias', 'model_ema.diffusion_modelinput_blocks50emb_layers1weight', 'model_ema.diffusion_modelinput_blocks50in_layers0bias', 'model_ema.diffusion_modelinput_blocks50in_layers0weight', 'model_ema.diffusion_modelinput_blocks50in_layers2bias', 'model_ema.diffusion_modelinput_blocks50in_layers2weight', 'model_ema.diffusion_modelinput_blocks50out_layers0bias', 'model_ema.diffusion_modelinput_blocks50out_layers0weight', 'model_ema.diffusion_modelinput_blocks50out_layers3bias', 'model_ema.diffusion_modelinput_blocks50out_layers3weight', 'model_ema.diffusion_modelinput_blocks51normbias', 'model_ema.diffusion_modelinput_blocks51normweight', 'model_ema.diffusion_modelinput_blocks51proj_inbias', 'model_ema.diffusion_modelinput_blocks51proj_inweight', 'model_ema.diffusion_modelinput_blocks51proj_outbias', 'model_ema.diffusion_modelinput_blocks51proj_outweight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0ffnet2bias', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0ffnet2weight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0norm1bias', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0norm1weight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0norm2bias', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0norm2weight', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0norm3bias', 'model_ema.diffusion_modelinput_blocks51transformer_blocks0norm3weight', 'model_ema.diffusion_modelinput_blocks60opbias', 'model_ema.diffusion_modelinput_blocks60opweight', 'model_ema.diffusion_modelinput_blocks70emb_layers1bias', 'model_ema.diffusion_modelinput_blocks70emb_layers1weight', 'model_ema.diffusion_modelinput_blocks70in_layers0bias', 'model_ema.diffusion_modelinput_blocks70in_layers0weight', 'model_ema.diffusion_modelinput_blocks70in_layers2bias', 'model_ema.diffusion_modelinput_blocks70in_layers2weight', 'model_ema.diffusion_modelinput_blocks70out_layers0bias', 'model_ema.diffusion_modelinput_blocks70out_layers0weight', 'model_ema.diffusion_modelinput_blocks70out_layers3bias', 'model_ema.diffusion_modelinput_blocks70out_layers3weight', 'model_ema.diffusion_modelinput_blocks70skip_connectionbias', 'model_ema.diffusion_modelinput_blocks70skip_connectionweight', 'model_ema.diffusion_modelinput_blocks71normbias', 'model_ema.diffusion_modelinput_blocks71normweight', 'model_ema.diffusion_modelinput_blocks71proj_inbias', 'model_ema.diffusion_modelinput_blocks71proj_inweight', 'model_ema.diffusion_modelinput_blocks71proj_outbias', 'model_ema.diffusion_modelinput_blocks71proj_outweight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0ffnet2bias', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0ffnet2weight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0norm1bias', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0norm1weight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0norm2bias', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0norm2weight', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0norm3bias', 'model_ema.diffusion_modelinput_blocks71transformer_blocks0norm3weight', 'model_ema.diffusion_modelinput_blocks80emb_layers1bias', 'model_ema.diffusion_modelinput_blocks80emb_layers1weight', 'model_ema.diffusion_modelinput_blocks80in_layers0bias', 'model_ema.diffusion_modelinput_blocks80in_layers0weight', 'model_ema.diffusion_modelinput_blocks80in_layers2bias', 'model_ema.diffusion_modelinput_blocks80in_layers2weight', 'model_ema.diffusion_modelinput_blocks80out_layers0bias', 'model_ema.diffusion_modelinput_blocks80out_layers0weight', 'model_ema.diffusion_modelinput_blocks80out_layers3bias', 'model_ema.diffusion_modelinput_blocks80out_layers3weight', 'model_ema.diffusion_modelinput_blocks81normbias', 'model_ema.diffusion_modelinput_blocks81normweight', 'model_ema.diffusion_modelinput_blocks81proj_inbias', 'model_ema.diffusion_modelinput_blocks81proj_inweight', 'model_ema.diffusion_modelinput_blocks81proj_outbias', 'model_ema.diffusion_modelinput_blocks81proj_outweight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0ffnet2bias', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0ffnet2weight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0norm1bias', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0norm1weight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0norm2bias', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0norm2weight', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0norm3bias', 'model_ema.diffusion_modelinput_blocks81transformer_blocks0norm3weight', 'model_ema.diffusion_modelinput_blocks90opbias', 'model_ema.diffusion_modelinput_blocks90opweight', 'model_ema.diffusion_modelmiddle_block0emb_layers1bias', 'model_ema.diffusion_modelmiddle_block0emb_layers1weight', 'model_ema.diffusion_modelmiddle_block0in_layers0bias', 'model_ema.diffusion_modelmiddle_block0in_layers0weight', 'model_ema.diffusion_modelmiddle_block0in_layers2bias', 'model_ema.diffusion_modelmiddle_block0in_layers2weight', 'model_ema.diffusion_modelmiddle_block0out_layers0bias', 'model_ema.diffusion_modelmiddle_block0out_layers0weight', 'model_ema.diffusion_modelmiddle_block0out_layers3bias', 'model_ema.diffusion_modelmiddle_block0out_layers3weight', 'model_ema.diffusion_modelmiddle_block1normbias', 'model_ema.diffusion_modelmiddle_block1normweight', 'model_ema.diffusion_modelmiddle_block1proj_inbias', 'model_ema.diffusion_modelmiddle_block1proj_inweight', 'model_ema.diffusion_modelmiddle_block1proj_outbias', 'model_ema.diffusion_modelmiddle_block1proj_outweight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0ffnet2bias', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0ffnet2weight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0norm1bias', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0norm1weight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0norm2bias', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0norm2weight', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0norm3bias', 'model_ema.diffusion_modelmiddle_block1transformer_blocks0norm3weight', 'model_ema.diffusion_modelmiddle_block2emb_layers1bias', 'model_ema.diffusion_modelmiddle_block2emb_layers1weight', 'model_ema.diffusion_modelmiddle_block2in_layers0bias', 'model_ema.diffusion_modelmiddle_block2in_layers0weight', 'model_ema.diffusion_modelmiddle_block2in_layers2bias', 'model_ema.diffusion_modelmiddle_block2in_layers2weight', 'model_ema.diffusion_modelmiddle_block2out_layers0bias', 'model_ema.diffusion_modelmiddle_block2out_layers0weight', 'model_ema.diffusion_modelmiddle_block2out_layers3bias', 'model_ema.diffusion_modelmiddle_block2out_layers3weight', 'model_ema.diffusion_modelout0bias', 'model_ema.diffusion_modelout0weight', 'model_ema.diffusion_modelout2bias', 'model_ema.diffusion_modelout2weight', 'model_ema.diffusion_modeloutput_blocks00emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks00emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks00in_layers0bias', 'model_ema.diffusion_modeloutput_blocks00in_layers0weight', 'model_ema.diffusion_modeloutput_blocks00in_layers2bias', 'model_ema.diffusion_modeloutput_blocks00in_layers2weight', 'model_ema.diffusion_modeloutput_blocks00out_layers0bias', 'model_ema.diffusion_modeloutput_blocks00out_layers0weight', 'model_ema.diffusion_modeloutput_blocks00out_layers3bias', 'model_ema.diffusion_modeloutput_blocks00out_layers3weight', 'model_ema.diffusion_modeloutput_blocks00skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks00skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks100emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks100emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks100in_layers0bias', 'model_ema.diffusion_modeloutput_blocks100in_layers0weight', 'model_ema.diffusion_modeloutput_blocks100in_layers2bias', 'model_ema.diffusion_modeloutput_blocks100in_layers2weight', 'model_ema.diffusion_modeloutput_blocks100out_layers0bias', 'model_ema.diffusion_modeloutput_blocks100out_layers0weight', 'model_ema.diffusion_modeloutput_blocks100out_layers3bias', 'model_ema.diffusion_modeloutput_blocks100out_layers3weight', 'model_ema.diffusion_modeloutput_blocks100skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks100skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks101normbias', 'model_ema.diffusion_modeloutput_blocks101normweight', 'model_ema.diffusion_modeloutput_blocks101proj_inbias', 'model_ema.diffusion_modeloutput_blocks101proj_inweight', 'model_ema.diffusion_modeloutput_blocks101proj_outbias', 'model_ema.diffusion_modeloutput_blocks101proj_outweight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0ffnet2bias', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0ffnet2weight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0norm1bias', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0norm1weight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0norm2bias', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0norm2weight', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0norm3bias', 'model_ema.diffusion_modeloutput_blocks101transformer_blocks0norm3weight', 'model_ema.diffusion_modeloutput_blocks10emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks10emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks10in_layers0bias', 'model_ema.diffusion_modeloutput_blocks10in_layers0weight', 'model_ema.diffusion_modeloutput_blocks10in_layers2bias', 'model_ema.diffusion_modeloutput_blocks10in_layers2weight', 'model_ema.diffusion_modeloutput_blocks10out_layers0bias', 'model_ema.diffusion_modeloutput_blocks10out_layers0weight', 'model_ema.diffusion_modeloutput_blocks10out_layers3bias', 'model_ema.diffusion_modeloutput_blocks10out_layers3weight', 'model_ema.diffusion_modeloutput_blocks10skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks10skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks110emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks110emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks110in_layers0bias', 'model_ema.diffusion_modeloutput_blocks110in_layers0weight', 'model_ema.diffusion_modeloutput_blocks110in_layers2bias', 'model_ema.diffusion_modeloutput_blocks110in_layers2weight', 'model_ema.diffusion_modeloutput_blocks110out_layers0bias', 'model_ema.diffusion_modeloutput_blocks110out_layers0weight', 'model_ema.diffusion_modeloutput_blocks110out_layers3bias', 'model_ema.diffusion_modeloutput_blocks110out_layers3weight', 'model_ema.diffusion_modeloutput_blocks110skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks110skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks111normbias', 'model_ema.diffusion_modeloutput_blocks111normweight', 'model_ema.diffusion_modeloutput_blocks111proj_inbias', 'model_ema.diffusion_modeloutput_blocks111proj_inweight', 'model_ema.diffusion_modeloutput_blocks111proj_outbias', 'model_ema.diffusion_modeloutput_blocks111proj_outweight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0ffnet2bias', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0ffnet2weight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0norm1bias', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0norm1weight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0norm2bias', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0norm2weight', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0norm3bias', 'model_ema.diffusion_modeloutput_blocks111transformer_blocks0norm3weight', 'model_ema.diffusion_modeloutput_blocks20emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks20emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks20in_layers0bias', 'model_ema.diffusion_modeloutput_blocks20in_layers0weight', 'model_ema.diffusion_modeloutput_blocks20in_layers2bias', 'model_ema.diffusion_modeloutput_blocks20in_layers2weight', 'model_ema.diffusion_modeloutput_blocks20out_layers0bias', 'model_ema.diffusion_modeloutput_blocks20out_layers0weight', 'model_ema.diffusion_modeloutput_blocks20out_layers3bias', 'model_ema.diffusion_modeloutput_blocks20out_layers3weight', 'model_ema.diffusion_modeloutput_blocks20skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks20skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks21convbias', 'model_ema.diffusion_modeloutput_blocks21convweight', 'model_ema.diffusion_modeloutput_blocks30emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks30emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks30in_layers0bias', 'model_ema.diffusion_modeloutput_blocks30in_layers0weight', 'model_ema.diffusion_modeloutput_blocks30in_layers2bias', 'model_ema.diffusion_modeloutput_blocks30in_layers2weight', 'model_ema.diffusion_modeloutput_blocks30out_layers0bias', 'model_ema.diffusion_modeloutput_blocks30out_layers0weight', 'model_ema.diffusion_modeloutput_blocks30out_layers3bias', 'model_ema.diffusion_modeloutput_blocks30out_layers3weight', 'model_ema.diffusion_modeloutput_blocks30skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks30skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks31normbias', 'model_ema.diffusion_modeloutput_blocks31normweight', 'model_ema.diffusion_modeloutput_blocks31proj_inbias', 'model_ema.diffusion_modeloutput_blocks31proj_inweight', 'model_ema.diffusion_modeloutput_blocks31proj_outbias', 'model_ema.diffusion_modeloutput_blocks31proj_outweight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0ffnet2bias', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0ffnet2weight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0norm1bias', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0norm1weight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0norm2bias', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0norm2weight', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0norm3bias', 'model_ema.diffusion_modeloutput_blocks31transformer_blocks0norm3weight', 'model_ema.diffusion_modeloutput_blocks40emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks40emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks40in_layers0bias', 'model_ema.diffusion_modeloutput_blocks40in_layers0weight', 'model_ema.diffusion_modeloutput_blocks40in_layers2bias', 'model_ema.diffusion_modeloutput_blocks40in_layers2weight', 'model_ema.diffusion_modeloutput_blocks40out_layers0bias', 'model_ema.diffusion_modeloutput_blocks40out_layers0weight', 'model_ema.diffusion_modeloutput_blocks40out_layers3bias', 'model_ema.diffusion_modeloutput_blocks40out_layers3weight', 'model_ema.diffusion_modeloutput_blocks40skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks40skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks41normbias', 'model_ema.diffusion_modeloutput_blocks41normweight', 'model_ema.diffusion_modeloutput_blocks41proj_inbias', 'model_ema.diffusion_modeloutput_blocks41proj_inweight', 'model_ema.diffusion_modeloutput_blocks41proj_outbias', 'model_ema.diffusion_modeloutput_blocks41proj_outweight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0ffnet2bias', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0ffnet2weight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0norm1bias', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0norm1weight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0norm2bias', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0norm2weight', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0norm3bias', 'model_ema.diffusion_modeloutput_blocks41transformer_blocks0norm3weight', 'model_ema.diffusion_modeloutput_blocks50emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks50emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks50in_layers0bias', 'model_ema.diffusion_modeloutput_blocks50in_layers0weight', 'model_ema.diffusion_modeloutput_blocks50in_layers2bias', 'model_ema.diffusion_modeloutput_blocks50in_layers2weight', 'model_ema.diffusion_modeloutput_blocks50out_layers0bias', 'model_ema.diffusion_modeloutput_blocks50out_layers0weight', 'model_ema.diffusion_modeloutput_blocks50out_layers3bias', 'model_ema.diffusion_modeloutput_blocks50out_layers3weight', 'model_ema.diffusion_modeloutput_blocks50skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks50skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks51normbias', 'model_ema.diffusion_modeloutput_blocks51normweight', 'model_ema.diffusion_modeloutput_blocks51proj_inbias', 'model_ema.diffusion_modeloutput_blocks51proj_inweight', 'model_ema.diffusion_modeloutput_blocks51proj_outbias', 'model_ema.diffusion_modeloutput_blocks51proj_outweight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0ffnet2bias', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0ffnet2weight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0norm1bias', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0norm1weight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0norm2bias', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0norm2weight', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0norm3bias', 'model_ema.diffusion_modeloutput_blocks51transformer_blocks0norm3weight', 'model_ema.diffusion_modeloutput_blocks52convbias', 'model_ema.diffusion_modeloutput_blocks52convweight', 'model_ema.diffusion_modeloutput_blocks60emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks60emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks60in_layers0bias', 'model_ema.diffusion_modeloutput_blocks60in_layers0weight', 'model_ema.diffusion_modeloutput_blocks60in_layers2bias', 'model_ema.diffusion_modeloutput_blocks60in_layers2weight', 'model_ema.diffusion_modeloutput_blocks60out_layers0bias', 'model_ema.diffusion_modeloutput_blocks60out_layers0weight', 'model_ema.diffusion_modeloutput_blocks60out_layers3bias', 'model_ema.diffusion_modeloutput_blocks60out_layers3weight', 'model_ema.diffusion_modeloutput_blocks60skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks60skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks61normbias', 'model_ema.diffusion_modeloutput_blocks61normweight', 'model_ema.diffusion_modeloutput_blocks61proj_inbias', 'model_ema.diffusion_modeloutput_blocks61proj_inweight', 'model_ema.diffusion_modeloutput_blocks61proj_outbias', 'model_ema.diffusion_modeloutput_blocks61proj_outweight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0ffnet2bias', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0ffnet2weight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0norm1bias', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0norm1weight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0norm2bias', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0norm2weight', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0norm3bias', 'model_ema.diffusion_modeloutput_blocks61transformer_blocks0norm3weight', 'model_ema.diffusion_modeloutput_blocks70emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks70emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks70in_layers0bias', 'model_ema.diffusion_modeloutput_blocks70in_layers0weight', 'model_ema.diffusion_modeloutput_blocks70in_layers2bias', 'model_ema.diffusion_modeloutput_blocks70in_layers2weight', 'model_ema.diffusion_modeloutput_blocks70out_layers0bias', 'model_ema.diffusion_modeloutput_blocks70out_layers0weight', 'model_ema.diffusion_modeloutput_blocks70out_layers3bias', 'model_ema.diffusion_modeloutput_blocks70out_layers3weight', 'model_ema.diffusion_modeloutput_blocks70skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks70skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks71normbias', 'model_ema.diffusion_modeloutput_blocks71normweight', 'model_ema.diffusion_modeloutput_blocks71proj_inbias', 'model_ema.diffusion_modeloutput_blocks71proj_inweight', 'model_ema.diffusion_modeloutput_blocks71proj_outbias', 'model_ema.diffusion_modeloutput_blocks71proj_outweight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0ffnet2bias', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0ffnet2weight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0norm1bias', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0norm1weight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0norm2bias', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0norm2weight', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0norm3bias', 'model_ema.diffusion_modeloutput_blocks71transformer_blocks0norm3weight', 'model_ema.diffusion_modeloutput_blocks80emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks80emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks80in_layers0bias', 'model_ema.diffusion_modeloutput_blocks80in_layers0weight', 'model_ema.diffusion_modeloutput_blocks80in_layers2bias', 'model_ema.diffusion_modeloutput_blocks80in_layers2weight', 'model_ema.diffusion_modeloutput_blocks80out_layers0bias', 'model_ema.diffusion_modeloutput_blocks80out_layers0weight', 'model_ema.diffusion_modeloutput_blocks80out_layers3bias', 'model_ema.diffusion_modeloutput_blocks80out_layers3weight', 'model_ema.diffusion_modeloutput_blocks80skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks80skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks81normbias', 'model_ema.diffusion_modeloutput_blocks81normweight', 'model_ema.diffusion_modeloutput_blocks81proj_inbias', 'model_ema.diffusion_modeloutput_blocks81proj_inweight', 'model_ema.diffusion_modeloutput_blocks81proj_outbias', 'model_ema.diffusion_modeloutput_blocks81proj_outweight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0ffnet2bias', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0ffnet2weight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0norm1bias', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0norm1weight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0norm2bias', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0norm2weight', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0norm3bias', 'model_ema.diffusion_modeloutput_blocks81transformer_blocks0norm3weight', 'model_ema.diffusion_modeloutput_blocks82convbias', 'model_ema.diffusion_modeloutput_blocks82convweight', 'model_ema.diffusion_modeloutput_blocks90emb_layers1bias', 'model_ema.diffusion_modeloutput_blocks90emb_layers1weight', 'model_ema.diffusion_modeloutput_blocks90in_layers0bias', 'model_ema.diffusion_modeloutput_blocks90in_layers0weight', 'model_ema.diffusion_modeloutput_blocks90in_layers2bias', 'model_ema.diffusion_modeloutput_blocks90in_layers2weight', 'model_ema.diffusion_modeloutput_blocks90out_layers0bias', 'model_ema.diffusion_modeloutput_blocks90out_layers0weight', 'model_ema.diffusion_modeloutput_blocks90out_layers3bias', 'model_ema.diffusion_modeloutput_blocks90out_layers3weight', 'model_ema.diffusion_modeloutput_blocks90skip_connectionbias', 'model_ema.diffusion_modeloutput_blocks90skip_connectionweight', 'model_ema.diffusion_modeloutput_blocks91normbias', 'model_ema.diffusion_modeloutput_blocks91normweight', 'model_ema.diffusion_modeloutput_blocks91proj_inbias', 'model_ema.diffusion_modeloutput_blocks91proj_inweight', 'model_ema.diffusion_modeloutput_blocks91proj_outbias', 'model_ema.diffusion_modeloutput_blocks91proj_outweight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0ffnet2bias', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0ffnet2weight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0norm1bias', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0norm1weight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0norm2bias', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0norm2weight', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0norm3bias', 'model_ema.diffusion_modeloutput_blocks91transformer_blocks0norm3weight', 'model_ema.diffusion_modeltime_embed0bias', 'model_ema.diffusion_modeltime_embed0weight', 'model_ema.diffusion_modeltime_embed2bias', 'model_ema.diffusion_modeltime_embed2weight', 'model_ema.num_updates', 'posterior_log_variance_clipped', 'posterior_mean_coef1', 'posterior_mean_coef2', 'posterior_variance', 'sqrt_alphas_cumprod', 'sqrt_one_minus_alphas_cumprod', 'sqrt_recip_alphas_cumprod', 'sqrt_recipm1_alphas_cumprod'])
Segmentation fault (core dumped)
Steps to reproduce
- Clone repository:
git clone https://github.com/comfyanonymous/ComfyUI - Install requirements:
python3 -m pip install -r requirements.txt && python3 -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm5.4.2 - Execute:
python3 main.py
Use the web interface. Click on "Queue Prompt" button. Then wait for the checkpoint to load (properly?) and fail when starting "CLIP Text Encode (prompt)".
Investigation
I have tried searching for something related to comfyui and core dumps, but only this issue popped. I monitored my machine to check if it had something to do with resources and it does not seem so. Monitored via nvtop.
Image of nvtop resource usage:
Replicated - same issue, very slightly different set up: OS: Ubuntu 22.04.2 LTS (Jammy Jellyfish) GPU: AMD Radeon 6750 XT Python: 3.10.12
I have a similar issue with: OS: Arch Linux GPU: AMD 5700 XT Python: 3.11.3
However I did have this working previously with Python 3.10 and downgrading Pytorch to version 1.13 with ROCm. I can't explain why this fixes it.
Unfortunately now the URL for 1.13 from the Pytorch website seems to be Access Denied currently so downgrading isn't possible but regardless it'd be good to figure out what is going on here.
I believe I'm having this issue too.
I started this discussion, https://github.com/comfyanonymous/ComfyUI/discussions/1174 And then I decided to leave it running - after about 15min, I got:
15217 Segmentation fault (core dumped) python3 main.py --preview-method auto
GPU - AMD® Radeon rx 6800 xt CPU - AMD® Ryzen 9 5900x 12-core processor × 24 RAM - 32.0 GiB OS - Ubuntu 22.04.3 LTS
I have installed the nightly version of Rocm (because it got me this far, aka actually executes) - rocm5.6 And the latest AMD drivers for Ubuntu - amdgpu-install_5.6.50600-1_all
I downgraded Rocm to 5.4.2 and now I get the below when I try to render an image... ...I also created a 32gb swap partition (yea, its added to fstab) just in case as I know that was an issue on windows. I found out that Ubuntu will install a tiny 1.5gb swap partition if you have over 16gb of RAM. I have 32gb so my swap partition wasn't even installed. However, that didn't help.
Total VRAM 16368 MB, total RAM 32002 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon RX 6800 : native
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
Adding extra search path checkpoints /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/Stable-diffusion
Adding extra search path configs /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/Stable-diffusion
Adding extra search path vae /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/VAE
Adding extra search path loras /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/Lora
Adding extra search path loras /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/LyCORIS
Adding extra search path upscale_models /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/ESRGAN
Adding extra search path upscale_models /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/RealESRGAN
Adding extra search path upscale_models /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/SwinIR
Adding extra search path embeddings /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/embeddings
Adding extra search path hypernetworks /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/hypernetworks
Adding extra search path controlnet /media/lethris/Cache/ASSETS/AI-TOOLS/MODEL-ASSETS/ControlNet
Starting server
To see the GUI go to: http://127.0.0.1:8188
got prompt
model_type EPS
adm 0
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
left over keys: dict_keys(['alphas_cumprod', 'alphas_cumprod_prev', 'betas', 'log_one_minus_alphas_cumprod', 'model_ema.decay', 'model_ema.num_updates', 'posterior_log_variance_clipped', 'posterior_mean_coef1', 'posterior_mean_coef2', 'posterior_variance', 'sqrt_alphas_cumprod', 'sqrt_one_minus_alphas_cumprod', 'sqrt_recip_alphas_cumprod', 'sqrt_recipm1_alphas_cumprod'])
./start.sh: line 9: 5745 Segmentation fault (core dumped) python3 main.py --preview-method auto
Okay I got this working again by downgrading to torch 1.13.1 + rocm 5.2 + python 3.10.
Earlier I said the pytorch extra url installation method (e.g pip install torch==1.13.1+rocm5.2 torchvision==0.14.1+rocm5.2 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/rocm5.2 ) wasn't working, however you can manually navigate to the .whl files, download them and then pip install torch-xxx.whl.
The URL I used is here: https://download.pytorch.org/whl/torch/ and I installed torch/torchaudio/torchvision this way (to the exact versions specified in the above extra url one liner)
An extra step I did was to install a virtualenv using Python 3.10 using python3.10 -m venv venv rather than 3.11 since it doesn't work with 1.13.1. All other dependencies were done with the standard pip install -r requirements.txt
Finally, I've always had to run ComfyUI with the --force-fp32 flag for my 5700 xt card, the final launch command is: HSA_OVERRIDE_GFX_VERSION=10.3.0 python main.py --force-fp32
Ofcourse it's not a fix for the Segmentation fault on the latest version but at least it can get us back to a working setup.
Earlier I said the pytorch extra url installation method (e.g
pip install torch==1.13.1+rocm5.2 torchvision==0.14.1+rocm5.2 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/rocm5.2) wasn't working, however you can manually navigate to the .whl files, download them and thenpip install torch-xxx.whl.
Hi! Thank you very much, @mikepjb ,I managed to get it working by downgrading it too. The pip install command worked for me perfectly fine (pip install torch==1.13.1+rocm5.2 torchvision==0.14.1+rocm5.2 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/rocm5.2 ), so no manually dealing with wheels.
Issue is still not solved, as this is a workaround to get it working and production code still has this issue.
Unfortunately for me, when I downgraded, it got as far as starting to render, then the monitor switched off.
Okay I got this working again by downgrading to torch 1.13.1 + rocm 5.2 + python 3.10.
@mikepjb may I please ask how exactly you got rocm 5.2 installed on arch?
I'm also using a 5700 on arch and have ran into the issues described: python 3.11.x not working with 5000 series gpu's pytorch 2.x breaks the gfx override needed for gpu's not officially rocm supported resulting in the coredump the pip install command seems to not grab some of the older packages
(Sorry for the derail, this is the only source I've seen that reported this setup as working)
Earlier I said the pytorch extra url installation method (e.g
pip install torch==1.13.1+rocm5.2 torchvision==0.14.1+rocm5.2 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/rocm5.2) wasn't working, however you can manually navigate to the .whl files, download them and thenpip install torch-xxx.whl.Hi! Thank you very much, @mikepjb ,I managed to get it working by downgrading it too. The pip install command worked for me perfectly fine (
pip install torch==1.13.1+rocm5.2 torchvision==0.14.1+rocm5.2 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/rocm5.2), so no manually dealing with wheels.Issue is still not solved, as this is a workaround to get it working and production code still has this issue.
This also worked for me. Thank you! Ubuntu 22.04 rocm 5.7 RX6800XT
I had the same issue which was resolved by following these steps :
- Cloning of this repository
- Creation of a virtual environment with python 3.11.
sudo python3.11 -m venv "py311amd" - Activation of the virtual environment
- Installation of requirements.
pip install requirements.txt - Launching ComfyUI and the default workflow --> Segmentation fault
- Back to the virtual environement, setting hsa_override_gfx_version to 10.3.0.
export hsa_override_gfx_version=10.3.0 - Launching ComfyUI again and then boom, it works
With theses steps, I can use the latest stable pytorch version, which is compiled against rocm 5.6. Works with a 6700 XT, on Kubuntu 22.04.3 LTS. By the way, I have rocm 5.7.1 installed system-wide. I don't know if this influences ComfyUI, just thought the info could be useful.
Hey i seem to have the same problem im currently downgrading my pytorch version by using 'pip install torch==1.13.1+rocm5.2 torchvision==0.14.1+rocm5.2 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/rocm5.2' i will let you know if that fixes it. i am on pop_os using an amd gpu
I have had this error for months. I recently realized that I had forgotten to add myself to the video and render groups at some point when I uninstalled/reinstalled ROCm and the AMD GPU stack.
I reinstalled all of the AMD/ROCm business again, making sure to add myself to the video and render groups, then I reinstalled comfyUI, and it works now.
