equinox icon indicating copy to clipboard operation
equinox copied to clipboard

Added "process_heads" to MultiheadAttention

Open Artur-Galstyan opened this issue 5 months ago • 1 comments

While checking this PR #568, I noticed that the "process_heads" part actually shouldn't be part of the RoPE embeddings PR as it's a separate thing. In theory, you could process the heads in any way you want.

Therefore, I thought it'd be best to make the PRs into smaller, more manageable chunks.

Artur-Galstyan avatar Jan 30 '24 10:01 Artur-Galstyan