LoRA
LoRA copied to clipboard
microsoft
Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"
Hi, I made some changes to the layers.py, however on running the mrpc task on roberta_base, it seems that __init__() and forward() of the linear layers is not being called...
Hi, Thanks for the great work. I am trying to reproduce the result of Roberta-large on the NLU tasks, however, I got a CoLA score = 0 and MNLI =...
My steps: ``` git clone https://github.com/microsoft/LoRA.git cd LoRA pip install -e . cd examples/NLU pip install -e . ``` Change `export num_gpus=8` to `export num_gpus=1` in `roberta_large_cola.sh` Then `CUDA_VISIBLE_DEVICES=0 bash...
in the table 3 of the paper "LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS" it says that finetuning top2 layer of GPT2 meidium require 25.19M trainable parameters. how to...
The recent commit https://github.com/microsoft/LoRA/commit/a0a92e0f26c067cf94747bdbf1ce73793fa44d19 flipped `A` and `B` in the comment for the LoRA `Linear` module: https://github.com/microsoft/LoRA/blob/a0a92e0f26c067cf94747bdbf1ce73793fa44d19/loralib/layers.py#L119-L125 The LoRA `Embedding` module similarly has the initialization flipped (not sure if this...
Hi Authors, I want to reproduce the results but the estimated performances in the paper come from several different random seeds. Can I know how you determined the random seed...
In the following code, the comment says weight A is init as usual (kaiming init as in other places in the code) and B as zeros. However, the behavior is...
I want to reproduce the performance of RoBERTa base Fine-tune. Modifying the apply_lora from True to False in the roberta_base_cola.sh code does not produce proper performance. What else should I...
