Incorporating Corrected Implementation of ROME in EasyEdit

[akshat57]

Hi,

Recent works found some instability when using ROME resulting in model collapse. We found that the original implementation of ROME was not the correct representation of update equation derived in the paper, and we've created a repo with the corrected implementation of ROME, a version we're calling r-ROME.

We've explained in the paper the exact details of the irregularities in the original implementation in our paper here - https://arxiv.org/abs/2403.07175

We also find that the corrected implementation has better overall performance and better generalization and localization, and also allows for stable sequential editing without model collapse.

We also provide the corrected code here - https://github.com/scalable-model-editing/rebuilding-rome

We would be grateful if our work can be incorporated in your repository and framework. We also believe it will add value to the EasyEdit community.

Thank, Akshat

[zxlzr]

Dear Akshat,

Thanks for your great work.

We will add r-ROME in EasyEdit.

We are recently rushing for the NeurIPS deadline and might need some time to integrate. If you are interested, you can also submit a PR.

Best,

Team of EasyEdit

[akshat57]

Will do. Thanks so much. Will reach out here when we submit a PR.

[sidnb13]

Just submitted the PR to implement R-ROME: #237

[pengzju]

Thanks for your contribution, I will reorganize your code and rename it to r-rome to retain the original implementation of rome.

[akshat57]

Hi!

Again, thanks so much for considering our work to be part of the easyedit offering!

Wanted to check in to see if there is an update or an estimated timeline for incorporating r-ROME into Easyedit?

[pengzju]

Neurips submission is coming soon. I will restart this matter after 5.22 and support it as soon as possible. Thank you for your timely follow-up. If you are interested, I also welcome you to submit a new PR. 😊

[piotrmigdalek]

Hi, is the R-ROME implementation supporting llama2-7b and gpt-j as in hparams or basically all the models as stated in README?

[pengzju]

I added a standalone implementation of R-ROME based on @akshat57 's implementation. In theory, it supports any models that are compatible with ROME, as there were no significant code changes.

[piotrmigdalek]

Thanks. Though I couldn't run anything in fp16 format as error in matrix multiplication occured.

[pengzju]

I have fixed this issue. Please update the code and try it again

[zxlzr]

Thanks. Though I couldn't run anything in fp16 format as error in matrix multiplication occured.

Sorry for the late reply.