xlabs Flux controlnet implementation.

[comfyanonymous]

I'm leaving this here because I'm not sure if I implemented it correctly or if the results are supposed to be this bad.

It only seems to work with FluxGuidance set to 4.0

Here's a workflow for anyone who wants to test it:

You can find the model file here: https://huggingface.co/XLabs-AI/flux-controlnet-canny/tree/main

[gradient-diffusion]

we trained it with only gs 4 and tomorrow you will se better version

[huchenlei]

Playwright test failed for missing from __future__ import annotations.

[starinskycc]

https://huggingface.co/xinsir/controlnet-union-sdxl-1.0 Can we train a versatile ControlNET like this expert?

[meimeilook]

It works on controlnet weight 0.9 above, just works. waiting for better quality.

[ivanoff13]

[sunyiwk]

I have the same problem

[oumad]

The instantX version ~seems to have a more permissive license~ : https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny-alpha Would the same implementation work for it too ?

[JorgeR81]

InstantX is so large ( 7 GB ) in comparison with XLabs-AI ( 1.5 GB ) ! It may not work for lower end systems which can currently run Flux. I hope we can get good controlnet quality without the need for these sizes.

[doggeddalle]

The instantX version seems to have a more permissive license : https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny-alpha Would the same implementation work for it too ?

thanks for linking these so we know about them. browsing internet feels like fugue state these days

[JorgeR81]

It works on controlnet weight 0.9 above, just works. waiting for better quality.

The pose seems good, but it seems it got more of an illustration style, with the ControlNet.

@meimeilook Can you try adding their realism lora ?

https://huggingface.co/XLabs-AI/flux-RealismLora

[kaykay-drills]

Hi guys, thanks for the amazing stuff you are constantly pulling off!! Tried to pull this branch locally and I still get the error reported above :(

Error occurred when executing ControlNetLoader:

'NoneType' object has no attribute 'keys'

Will this one be merged tomorrow or in the upcoming week(s)?

[nikolaiusa]

Попробовал вытащить эту ветку локально, но все равно получаю ошибку, указанную выше :(

Error occurred when executing ControlNetLoader:

'NoneType' object has no attribute 'keys'

Будет ли это объединено завтра или в течение следующих недель?

+1

[doggeddalle]

Tried to pull this branch locally and I still get the error reported above :(

Error occurred when executing ControlNetLoader:

'NoneType' object has no attribute 'keys'

Will this one be merged tomorrow or in the upcoming week(s)?

gonna assume its borked

[nikolaiusa]

git checkout xlabs_flux_controlnet

[kaykay-drills]

git checkout xlabs_flux_controlnet

yes, first thing I did was to checkout this branch :) still doesn't work :(

F:\Projects\AI\COMFY-UI\ComfyUI_windows_portable\ComfyUI\ComfyUI>git branch
* xlabs_flux_controlnet

!!! Exception during processing!!! 'NoneType' object has no attribute 'keys'

[ZiBrianQian]

git checkout xlabs_flux_controlnet

yes, first thing I did was to checkout this branch :) still doesn't work :(

F:\Projects\AI\COMFY-UI\ComfyUI_windows_portable\ComfyUI\ComfyUI>git branch
* xlabs_flux_controlnet

!!! Exception during processing!!! 'NoneType' object has no attribute 'keys'

Update ComfyUI. then run the git checkout xlabs_flux_controlnet again. It worked for m.

[Bortus-AI]

git checkout xlabs_flux_controlnet

yes, first thing I did was to checkout this branch :) still doesn't work :(

F:\Projects\AI\COMFY-UI\ComfyUI_windows_portable\ComfyUI\ComfyUI>git branch
* xlabs_flux_controlnet

!!! Exception during processing!!! 'NoneType' object has no attribute 'keys'

Update ComfyUI. then run the git checkout xlabs_flux_controlnet again. It worked for m.

that worked!

[WingeD123]

InstantX is so large ( 7 GB ) in comparison with XLabs-AI ( 1.5 GB ) ! It may not work for lower end systems which can currently run Flux. I hope we can get good controlnet quality without the need for these sizes.

InstantX just replaced the 7g one with a 3.3g version

[kaykay-drills]

F:\Projects\AI\COMFY-UI\ComfyUI_windows_portable\ComfyUI\ComfyUI>git branch
xlabs_flux_controlnet

This Morning I updated ComfyUI. The previous New error spotted:

'NoneType' object has no attribute 'lower'

Quick update: I tried to merge master locally into this branch as well but no luck

[ZiBrianQian]

F:\Projects\AI\COMFY-UI\ComfyUI_windows_portable\ComfyUI\ComfyUI>git branch
xlabs_flux_controlnet

New error spotted:

'NoneType' object has no attribute 'lower'

Quick update: I tried to merge master locally into this branch as well but no luck

Had the same problem, with XLabs Flux_dev_fp8. Try this model instead https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors

[kaykay-drills]

Had the same problem, with XLabs Flux_dev_fp8. Try this model instead https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors

Now the error is on the SampleCustomAdvanced

TypeError: ControlNetFlux.forward() missing 1 required positional argument: 'y'

[93041025]

The instantX version ~seems to have a more permissive license~ : https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny-alpha Would the same implementation work for it too ?

it generates another error "Error occurred when executing ControlNetLoader:

'pos_embed.proj.weight'"

[djdookie]

After some testing I can say this branch + controlnet works great for me even on 1024x1024. Thanks for the quick implementation.

[BaNZaY777666]

i have slightly different problem( Using Stability Matrix though.

Error occurred when executing ControlNetLoader:

'pos_embed.proj.weight'

File "C:\AI\Matrix\Packages\ComfyUI\execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "C:\AI\Matrix\Packages\ComfyUI\execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "C:\AI\Matrix\Packages\ComfyUI\execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "C:\AI\Matrix\Packages\ComfyUI\nodes.py", line 720, in load_controlnet controlnet = comfy.controlnet.load_controlnet(controlnet_path) File "C:\AI\Matrix\Packages\ComfyUI\comfy\controlnet.py", line 491, in load_controlnet return load_controlnet_mmdit(controlnet_data) File "C:\AI\Matrix\Packages\ComfyUI\comfy\controlnet.py", line 404, in load_controlnet_mmdit new_sd = comfy.model_detection.convert_diffusers_mmdit(sd, "") File "C:\AI\Matrix\Packages\ComfyUI\comfy\model_detection.py", line 500, in convert_diffusers_mmdit depth = state_dict["pos_embed.proj.weight"].shape[0] // 64

[kaykay-drills]

'pos_embed.proj.weight'

this is because you are using the The InstantX version. This branch covers xlabs controlnet model only, I believe.

[BaNZaY777666]

'pos_embed.proj.weight'

this is because you are using the The InstantX version. This branch covers xlabs controlnet model only, I believe.

Indeed. But xlabs controlnet not doing well. Or I am doing something wrong?

[kaykay-drills]

Indeed. But xlabs controlnet not doing well. Or I am doing something wrong?

Please refer to this comment, it solved my issue.

Had the same problem, with XLabs Flux_dev_fp8. Try this model instead https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors

[BaNZaY777666]

Indeed. But xlabs controlnet not doing well. Or I am doing something wrong?

Please refer to this comment, it solved my issue.

Had the same problem, with XLabs Flux_dev_fp8. Try this model instead https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors

Still kinda bad

[kaykay-drills]

Still kinda bad

Let's be positive! it works also for you! Great success!

Also, try to use the nodes provided in the image proposed. at the very top.

[kaykay-drills]

It works! Thanks for the help!

A few points:

• Lora flux_realism_lora can work along with Flux CNet
• SplitSigma node is not supported
• ClipTextEncodeFlux is also not supported (that was my issue)
• applying this xlabs_flux_controlnet CNet renders weird lines to the output image with CFG 3.5 tho it shines with CFG4