How can I use LoRA to fine tune ESMC model?

[hassanzadehmahdi]

I want to use ESMC for getting the embedding of the protein and I also want to use LoRA to adapt the model to my specific task. How can I train this LoRA(base model should freeze) with a MLP for classification on top of it?

[peiyaoli]

same question here

[lhallee]

Hi @hassanzadehmahdi and @peiyaoli,

I wanted to mention that my group has implemented ESMC in a fully Huggingface compatible way (called ESM++). This means you can use the Peft package to finetune them with LoRA. Just pay attention to the ESM licenses before you train anything. You can find the implementation here with some other goodies as well :smile: . Best, Logan

[ShaowenJCornell]

Have you solved this issue? Or using the ESM++ now?