axon icon indicating copy to clipboard operation
axon copied to clipboard

Published 20 hours ago • verified publisher icon elixir-nx

Add attention layers

Open seanmor5 opened this issue 3 years ago • 2 comments

We should work out a common API and implementation for working with attention layers. Right now they are easily implemented as custom layers, but we should have conveniences to save user's time

seanmor5 avatar Jun 07 '22 00:06 seanmor5

Can I take this on?

joaogui1 avatar Sep 11 '22 23:09 joaogui1

Yes, definitely! Let me know if you run into any issues or have questions

seanmor5 avatar Sep 12 '22 00:09 seanmor5

See https://github.com/elixir-nx/bumblebee

Maybe later it will make sense to move those attention implementations but for now it's okay :)

seanmor5 avatar Dec 09 '22 02:12 seanmor5