panndas icon indicating copy to clipboard operation
panndas copied to clipboard

develop attention-only transformers examples

Open charlesfrye opened this issue 3 years ago • 0 comments

This paper provides a mathematical framework for thinking about attention-only transformers.

If we drop the softmax, this becomes a pretty solid demo for Transformers in panndas -- borrowing the details for the problem from Brandon Rohrer's Transformers tutorial.

charlesfrye avatar Mar 27 '22 01:03 charlesfrye