vladmandic
[Feature]: Support for Stable Cascade LoRAs and embeddings
Feature description
I've now implemented LoRA and embedding training support in OneTrainer (for now only on a separate branch https://github.com/Nerogar/OneTrainer/tree/stable_cascade) . But to actually use them, support in the inference tools is needed. I propose the following format:
LoRA:
Each unet key has a prefix lora_prior_unet_, followed by the base model key in diffusers format, then lora_down.weight, lora_up.weight and alpha
Each text encoder key has a prefix lora_prior_te_, followed by the base model key in diffusers format, then lora_down.weight, lora_up.weight and alpha
For the unet, I have only included lora weights for the attention blocks. But the naming scheme would be the same for other blocks.
Embedding:
The file has a single key: clip_g. This is the same format as SDXL embeddings, just without the clip_l key.
Example files
I have attached two example files. The example LoRA is for the 1B variant of stage C. Both are untrained, they are just examples to show the file format. examples.zip
For reference, I have opened the same issue for ComfyUI https://github.com/comfyanonymous/ComfyUI/issues/2831
Version Platform Description
No response
@Nerogar The standardized key names are much appreciated, as are the untrained samples.
@vladmandic I will have to familiarize myself with your implementation of Stable Cascade, and then we can discuss ways and means.
@Nerogar The standardized key names are much appreciated, as are the untrained samples.
@vladmandic I will have to familiarize myself with your implementation of Stable Cascade, and then we can discuss ways and means.
rights now its experimental at best and uses diffusers side branch, best to wait few days until diffusers work stabilizes. few days are not going to make big difference.
I've updated the example files to reflect the new format. unet attention keys are not saved in diffusers format anymore.
Sorry for all this back and forth, but the comfy dev found another issue with my lora keys. I updated the example file again. The only thing that changes is the key name for the out projection of unet attention blocks
No worries. And since you've cross-posted the issue, don't need to update here for each item.
sc lora support via peft methods is added for a while now. there is no current plans to add it via native methods.