Does this work with IPAdapter node?

[ynie]

See what I have above:

I tried to put the encoder node before and after the IPAdapters, It seems to have heavy impact on the image rendering. I also tried Attention Injection as well, it got worse.. what did I do wrong? Thanks!

[huchenlei]

Tested that it works for IP-Adapter. Just make sure your input to IPAdapter should ideally have simple background or transparent background so that there are not too much noise information from the background getting captured.

ld_ipadapter.json

[ynie]

Got it. Thank you!

[ynie]

Actually, just to follow your example, has the attention injection ever work for you? If I choose to use attention injection, the rendering seems broken to me, which is very different than the author mentioned here:

https://github.com/layerdiffusion/sd-forge-layerdiffuse However, in practice, I find the layer_xl_transparent_attn.safetensors will lead to better results. This layer_xl_transparent_conv.safetensors is still included for some special use cases that needs special prompt understanding. Also, this model may introduce a strong style influence to the base model.

[huchenlei]

I can confirm that using attn injection is going to yield worse results. The reason behind is that IPAdapter also injects attn blocks, so there might be conflict on attn weights trying to adapt to 2 tasks at the same time.

[ynie]

It does seem like using Conv will impact the rendering quality. I'm wondering if there's any better solution to this.