mend icon indicating copy to clipboard operation
mend copied to clipboard

Learning rate in MEND edit procedure

Open Xiaoxi-Luo-CL opened this issue 1 year ago • 0 comments

Hi, I have a small question about MEND edit procedure in test time: In your paper's Algorithm 2, line 7 (updating the weight matrix of layer $l$), you wrote $\tilde{W}l \gets W_l-\tilde{\delta}{l+1} \tilde{u}_l ^T$, and there is no learning rate in this equation. However, in mend.py, line 227, updates = {n: lr * g for lr, (n, g) in zip(self.edit_lrs, mean_grads.items())}, there actually is learning rate when updating, right?

Thank you!

Xiaoxi-Luo-CL avatar Dec 20 '24 11:12 Xiaoxi-Luo-CL