functional-transformer icon indicating copy to clipboard operation
functional-transformer copied to clipboard

Make value heads nonsquare and add back head concatenation

Open awf opened this issue 2 years ago • 0 comments

As noted in https://github.com/awf/functional-transformer/discussions/6 the model does not match the original code, or indeed the original transformer paper. I therefore consider this a "transformer variant", but of course it would be sensible to make it match and check if that improves/disimproves performance.

awf avatar Dec 12 '22 17:12 awf