T2I-Adapter, Documentation Improvement Request

[createchange]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

I found a thread on Reddit that describes the benefits of T2I-Adapters, summarized as such:

For controlnets the large (~1GB) controlnet model is run at every single iteration for both the positive and negative prompt which slows down generation time considerably and taking a bunch of memory.

For T2I-Adapter the ~300MB model is only run once in total at the beginning which means it has pretty much no effect on generation speed.

Thread here: https://www.reddit.com/r/StableDiffusion/comments/11don30/a_quick_comparison_between_controlnets_and/

As such, I started to look into how to utilize these. Following the instructions in this repository, I downloaded the models (I retrieved the .safetensor versions from here: https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main).

Here is where I encounter issues with understanding the documentation:

Copy corresponding config file and rename it to the same name as the model - see list below.

What is the corresponding config file? I see no .yaml files in the official TencentARC/T2I-Adapter HF repository, and only two in the HF repo linked above:

• cldm_v15.yaml
• cldm_v21.yaml

However, given the chart provided in the README:

Adapter	Config
t2iadapter_canny_sd14v1.pth	sketch_adapter_v14.yaml
t2iadapter_sketch_sd14v1.pth	sketch_adapter_v14.yaml
t2iadapter_seg_sd14v1.pth	image_adapter_v14.yaml
t2iadapter_keypose_sd14v1.pth	image_adapter_v14.yaml
t2iadapter_openpose_sd14v1.pth	image_adapter_v14.yaml
t2iadapter_color_sd14v1.pth	t2iadapter_color_sd14v1.yaml
t2iadapter_style_sd14v1.pth	t2iadapter_style_sd14v1.yaml

I don't see how those config files would be the relevant ones. Given the fact that the config files in the above chart have two different naming schemes ( sd14 vs. sd14v1), it would seem that config files can't be a copy and paste from a single config, but renamed...

So, where are these config files actually coming from?

Steps to reproduce the problem

1. Follow documentation
2. Get confused

What should have happened?

1. Documentation should provide more explicit directions on the process

Commit where the problem happens

N/A

What browsers do you use to access the UI ?

No response

Command Line Arguments

N/A

Console logs

N/A

Additional information

No response

[createchange]

I think I may have been being dense. Is a correct view of the provided chart?

For the following adapter: t2iadapter_canny_sd14v1.pth:

Copy + Rename: sketch_adapter_v14.yaml -> t2iadapter_canny_sd14v1.yaml

?

---

Edit: This appears to have worked. However, when running a prompt with canny pre-processor and corresponding canny T2I-Adapter model, I received the following error:

RuntimeError: Input type (MPSFloatType) and weight type (MPSHalfType) should be the same

I resolved by passing in a the --no-half parameter at AUTOMATIC1111 runtime.

Is my understanding all correct, and is this a suitable way to resolve things?

Finally, I don't see a row for the T2I-Adapter for depth - which config would be a suitable one to copy and rename? Trial and error shows me it might be image_adapter_v14.yaml.

[orfeaus]

Same question. I don't see any relevant yaml file on huggingface that I can rename.

[missionfloyd]

They're in the models folder. But yeah, that could be handled better.

I also notice that t2iadapter_depth_sd14v1.pth is missing from that list. Does it not need one?

[Lexcess]

As a slight side note to the original premise or your issue, there are far smaller 'difference only' versions of the original controlNet models here: https://civitai.com/models/9868/controlnet-pre-trained-difference-models they are still larger (600-700mb), but the size reduction really adds up.

I haven't noticed any significant difference in quality with the diff versions, and that is a complaint of ths T2i implementations.

[78Alpha]

image_adapter_v14.yaml is the one to rename

image_adapter_v14.yaml > t2iadapter_depth_sd15v2 for the SD15v2 version

[S0LTIS]

WHERE DID t2iadapter_color_sd14v1.yaml GO???????????????????

[S0LTIS]

They're in the models folder . But yeah, that could be handled better.

I also notice that t2iadapter_depth_sd14v1.pth is missing from that list. Does it not need one?

It is not there.

[kingsushi001]

WHERE DID t2iadapter_color_sd14v1.yaml GO???????????????????

I would like to know as well

[bobcate]

@Mikubill, would you clarify the issue of missing .yaml files? Do we need them for the t2i adapter? If so where can we get them?