ComfyUI icon indicating copy to clipboard operation
ComfyUI copied to clipboard

The flux dev model outputs pure black on 4090

Open alex13by opened this issue 1 year ago • 8 comments

Expected Behavior

Troubleshoot the flux-fp8-dev black graph

Actual Behavior

Running flux-fp8-dev e4m3fn on 4090 using Vincennes graphs will always be black, it still appears after disabling all plugins, but there is no such problem on schnell's model and other models. When a black image appears, an error message is found in the background::RuntimeWarning: invalid value encountered in cast img = Image.fromarray(np.clip(i, 0, 255).astype(np.uint8))

Steps to Reproduce

Black diagrams appear on any sample workflow for flux Venting diagrams, including torch2.4.0, cu12.4B33CD61, and recent versions

Debug Logs

Starting server
To see the GUI go to: http://127.0.0.1:8188
FETCH DATA from: D: \comfyUI windows_portable\comfyUI\custom nodes\comfyUI-Manager extension-node-map.json [DONEI
got prompt
Using xformers attention in VAE
Using xformers attention in VAE
model weight dtype torch.float8 e4m3fn, manual cast: torch.bfloat16
model type FLUX
Requested to load FluxClipModel
Loading 1 new model
loaded completely 0.0 4777.53759765625 True
clip missing:['text projection.weight']
D: \Comf
windows portable\comfyUI\comfy\ldm modules attention.py:407: UserWarning: 1Torch was not compiled with flash att
ion.(T
<iternally at c: lactions-runnerl work \pytorchlpytorch\builder windows \pytorch aten src\ATen\nativeltransformers\c>dp_uti
rigger
ls.cpp:555.)
out = torch.nn.functional.scaled dot product attention(q, k, v, attn mask=mask, dropout p=8.8, is causal=False)
Requested to load Flux
Loading 1 new model
loaded completely 0.0 11350.048889160156 True
6/6[00:02<00:00,2.32it/s]
180%
Requested to load AutoencodingEngine
oading 1 new model
loaded completely 0.0159.87335777282715 True
D:\ComfyUI windows portable\comfyUI\nodes.py:1498: RuntimeWarning: invalid value encountered in cast
img=Image.fromarray(np.clip(i,0,255).astype(np.uint8))
Prompt executed in 19.87 seconds

Other

No response

alex13by avatar Aug 29 '24 09:08 alex13by

Downgrade pytorch to 2.3.1 and try again.

ltdrdata avatar Aug 29 '24 10:08 ltdrdata

将 pytorch 降级到 2.3.1 并重试。

Same problem under torch2.31 cu121

alex13by avatar Aug 29 '24 12:08 alex13by

将 pytorch 降级到 2.3.1 并重试。 comfyui.log

This is the full log of the 2.4.0 run, including inference prompts, Before upgrading to 2.4.0 is torch 2.3.1+CU121

alex13by avatar Aug 30 '24 00:08 alex13by

I have the same problem.

hubin858130 avatar Aug 30 '24 08:08 hubin858130

linux,3090, pytorch version: 2.4.0+cu121,也出现同样的问题,print(torch.isnan(image).any()) 打印结果显示image是NAN值,具体怎么知道是在哪步出错了?

BuShangTou avatar Sep 04 '24 09:09 BuShangTou

将 pytorch 降级到 2.3.1 并重试。 comfyui.log

This is the full log of the 2.4.0 run, including inference prompts, Before upgrading to 2.4.0 is torch 2.3.1+CU121

What is your workflow? One of the typical cases where a black image appears is when selecting two t5 models instead of t5 + clip_l in the DualClipLoader.

ltdrdata avatar Sep 04 '24 14:09 ltdrdata

The most simplified workflow and CLIP selection is normal, if changed to 22G DEV original version using fp8 will not black plot, but using 11G fp8 version will black plot, this version of the model on other graphics cards did not find exceptions. https://huggingface.co/Kijai/flux-fp8

alex13by avatar Sep 05 '24 00:09 alex13by

My problem also occurred in CLIP. I used sd3_medium_incl_clips_t5xxlfp16.safetensors from stable-diffusion-3-medium. My workflow was sd3_medium_example_workflow_multi_prompt. The black image was caused by TripleCLIPLoader. I skipped using it and everything was fine. However, I don’t understand why this problem occurs on this graphics card and environment. There is no problem on other machines.

BuShangTou avatar Sep 05 '24 01:09 BuShangTou

I have the same problem. I hope there is a solution.

leezhenxue5099 avatar Sep 14 '24 14:09 leezhenxue5099

I have the same problem. I hope there is a solution.

i have the same problem while runs in docker image of flux dev...any update ? @leezhenxue5099

tzktok avatar Oct 04 '24 05:10 tzktok

i have the same problem while runs in docker image of flux dev...any update ?

adamwuyu avatar Dec 19 '24 05:12 adamwuyu

i have the same problem while runs in docker image of flux dev...any update ?

built with comfyui ?

tzktok avatar Dec 19 '24 05:12 tzktok