convert_checkpoint_to_lsg icon indicating copy to clipboard operation
convert_checkpoint_to_lsg copied to clipboard

Published 20 hours ago • verified publisher icon ccdv-ai

Convert DeBERTa to longDeBERTa

Open duyvuleo opened this issue 3 years ago • 1 comments
trafficstars

Hi,

Thanks for the great work.

Is it possible to convert DeBERTa models to longDeBERTa ones? Would you please help advise specific steps that I can follow?

Looking forward to your response. Thanks!

duyvuleo avatar Nov 03 '22 02:11 duyvuleo

Hi @duyvuleo

Currently, converting DeBERTa to Long DeBERTa is not possible because this model uses on a specific attention mecanism called "disentangled attention" which relies on different inputs + relative positional embedding.

To make DeBERTa compatible, some things need to be rethought specifically for this model. I may add DeBERTa in the future.

ccdv-ai avatar Nov 03 '22 09:11 ccdv-ai