LoRA
LoRA copied to clipboard
Parameter count on GPT-2 medium
According to the paper, the lora parameter count for GPT-2 medium is 0.35M, but since the hidden dimension is 1024, and the model have 24 layers, with a rank=4, the correct parameter count should be 1024 * 2[for B,A] * 4[for rank=4] * 2[for Q,V] * 24[num. of layers] = 393216 = 0.39M.
I also checked the code for the above calculation.
Do I miss something here?
Thank you very much!