EasyEdit Support for MALMEN

MALMEN: Massive Language Model Editing Network https://arxiv.org/abs/2311.04661 is the newest SOTA on Meta Learning method. In their experiments, it has the excellent performance with massive editing even better than MEMIT in certain aspects. I think it is a combination of MEND and MEMIT. May you add support for this method? Thank you! https://github.com/ChenmienTan/malmen

Dec 15 '23 07:12 Yzk1114

We will support MALMEN soon, thank you for your suggestions.

Dec 19 '23 02:12 zxlzr

Hi, the support for MALMEN is performing internal testing of the code and aiming for release within a month.

Mar 01 '24 06:03 zxlzr

I apologize for the delay. Currently, Easyedit has implemented partial support for the Malmen method, but due to framework compatibility issues, there are some performance compromises in certain functions. We will continue to improve support in the first half of the year. Thank you for your patience.

Mar 02 '24 07:03 xzwyyd

We apologize for the inconvenience.

Due to some compatibility issues between our framework and the original MALMEN code, we can only provide partial support at this time. We found that when the batch size is set to 1, the performance is relatively stable, but when the batch size is larger than 1, there will be a certain degree of performance drop.

After many attempts, we guessed that this problem may be related to our data loading process. However, given the complexity of modifying the framework and our resource constraints, we decided not to pursue further optimization in this area for the time being.

We appreciate your attention and valuable feedback on this issue. If you have any better solutions, you are warmly welcome to submit a PR to our GitHub repository.

Aug 21 '24 02:08 xzwyyd