For pretraining with LORA what is the expected output? A Lora Adapator or the complete pretrained model with adaptor merged in it?

[amitagh]

For pretraining with LORA what is the expected output? A Lora Adapator or the complete pretrained model with adaptor merged in it? I am pretraining LLM with 7B params with Lora.

[codemayq]

The direct product is LoRA adaptor. Then You can merge it into base model, like this https://github.com/hiyouga/LLaMA-Factory/tree/main/examples/merge_lora to get a complete model. bless.