mistral-inference icon indicating copy to clipboard operation
mistral-inference copied to clipboard

Published 20 hours ago • verified publisher icon mistralai

Q: Why rotary embedding applied only to queries and keys?

Open pmgautam opened this issue 10 months ago • 1 comments

In the codebase, rotary embeddings are applied only to queries and keys but not to values. Can someone point out to reasons/papers behind this design? Thank you in advance!!!

image

pmgautam avatar Apr 20 '24 08:04 pmgautam