invoke-ai
[bug]: ValueError: The SDXL LoRA could only be partially converted to diffusers format. converted=722, not_converted=560
Is there an existing issue for this problem?
- [X] I have searched the existing issues
Operating system
Windows
GPU vendor
Nvidia (CUDA)
GPU model
RTX 3060 TI
GPU VRAM
No response
Version number
3.6
Browser
Chrome
Python dependencies
{ "accelerate": "0.26.1", "compel": "2.0.2", "cuda": "12.1", "diffusers": "0.26.2", "numpy": "1.26.4", "opencv": "4.9.0.80", "onnx": "1.15.0", "pillow": "10.2.0", "python": "3.10.6", "torch": "2.1.2+cu121", "torchvision": "0.16.2+cu121", "transformers": "4.37.2", "xformers": "0.0.23.post1" }
What happened
Added SDXL Lora ad got this Message. Anyone know Solution?
Thanks
What you expected to happen
Image Generation with Lora
How to reproduce the problem
No response
Additional context
No response
Discord username
No response
This seems to be very dependent on the LoRA. Can you share which you tried to use?
Yep its Lora depending.
First i tried was Advanced Enhancer XL LoRA: https://civitai.com/models/124993/advanced-enhancer-xl-lora and second was: RMSDXL Enhance XL https://civitai.com/models/238442/rmsdxl-enhance-xl-tool
Other Loras are working
So I gave this one a bit of a poke myself after running into it - it looks like the cause is that those loras contain a set of layers named "lora_unet_up_blocks_1_attentions_2_". From what I can tell, these are already named in the huggingface format, and so don't need to be converted - and thus are generating the error.
I wrote a small test case for only generating the error if the key wasn't in either sd or hf format, and the lora loaded and seemed to have an effect - but I don't know enough about the whole underlying system to know if I just hid an issue that the error checking is supposed to prevent.
Thanks for the detective work. I’m not all that familiar with Lora loading myself, but I’ll dig into it and see if your fix is having the desired effect.
I've looked into this and I'm not sure what the right course of action is. There seems to be redundancy in the files. Consider Advanced Enhancer XL LoRA. It contains an SD key of lora_unet_output_blocks_0_1_transformer_blocks_0_attn1_to_k, which translates to an HD key of lora_unet_up_blocks_0_attentions_0_transformer_blocks_0_attn1_to_k.
Later on, the same file contains the HD key lora_unet_up_blocks_0_attentions_0_transformer_blocks_0_attn1_to_k , which is the same key that was converted previously. However, the value of this key is not the same. So if we simply use this key, it will overwrite the previous converted key with a different tensor, and there's no way of knowing which is the right one to use.
Unets have down/input layers, middle layers, and up/output layers. It doesn't make sense to represent the same layer twice. I'm strongly inclined to say that these LoRAs are broken and shouldn't be used.
What are your thoughts?
I've looked into this and I'm not sure what the right course of action is. There seems to be redundancy in the files. Consider Advanced Enhancer XL LoRA. It contains an SD key of
lora_unet_output_blocks_0_1_transformer_blocks_0_attn1_to_k, which translates to an HD key oflora_unet_up_blocks_0_attentions_0_transformer_blocks_0_attn1_to_k.Later on, the same file contains the HD key
lora_unet_up_blocks_0_attentions_0_transformer_blocks_0_attn1_to_k, which is the same key that was converted previously. However, the value of this key is not the same. So if we simply use this key, it will overwrite the previous converted key with a different tensor, and there's no way of knowing which is the right one to use.Unets have down/input layers, middle layers, and up/output layers. It doesn't make sense to represent the same layer twice. I'm strongly inclined to say that these LoRAs are broken and shouldn't be used.
What are your thoughts?
That sounds right - some training app has allowed generation of a "broken" LoRA. But I have to assume they are working in other UIs because the fallback behavior is just to let that layer get overridden?