Open-Sora-Plan icon indicating copy to clipboard operation
Open-Sora-Plan copied to clipboard

Published 20 hours ago • verified publisher icon PKU-YuanGroup

Can you guys try using rwkv instead of Transformers?

Open win10ogod opened this issue 11 months ago • 4 comments

Can you guys try using rwkv instead of Transformers?

win10ogod avatar Mar 17 '24 13:03 win10ogod

Hi, maybe it is a future work. Could you provide a PR for this?

LinB203 avatar Mar 20 '24 14:03 LinB203

Hi, maybe it is a future work. Could you provide a PR for this?

This is not my project but uses rwkv-v6 instead of Transformers. https://github.com/yynil/Di-R

win10ogod avatar Mar 21 '24 01:03 win10ogod

i think it's better to try mamba instead of rwkv, which is now considered as next gen architecture.

ZhengJun-AI avatar Mar 29 '24 11:03 ZhengJun-AI

i think it's better to try mamba instead of rwkv, which is now considered as next gen architecture.

But the performance is weaker! rwkv v6 is obviously more powerful than mamba!

win10ogod avatar Mar 29 '24 11:03 win10ogod