litgpt icon indicating copy to clipboard operation
litgpt copied to clipboard

Published 20 hours ago • verified publisher icon Lightning-AI

Can we integrating LongRoPE.

Open PSanni opened this issue 11 months ago • 2 comments

They have implemented LongRope patch for llama and mistral. Is it possible to port same into ligGPT ?

https://github.com/datamllab/LongLM/tree/master

PSanni avatar Feb 29 '24 07:02 PSanni

Hah, I was just bringing this up to my colleagues yesterday. I think it might be a nice addition in the future.

rasbt avatar Feb 29 '24 14:02 rasbt

The Git repo you provided, LongLM, appears to be an implementation of Self-Extend, not LongRoPE. Did I misunderstand something?

ryan-minato avatar Jun 07 '24 08:06 ryan-minato