annotated_deep_learning_paper_implementations icon indicating copy to clipboard operation
annotated_deep_learning_paper_implementations copied to clipboard
verified publisher icon labmlai

Request - performers

Open johndpope opened this issue 4 years ago • 4 comments

https://www.youtube.com/watch?v=xJrKIPwVwGM&t=767s

https://ai.googleblog.com/2020/10/rethinking-attention-with-performers.html

johndpope avatar Feb 08 '21 18:02 johndpope

Hi, we are a little busy these days rewriting https://github.com/lab-ml/app and I'm down with fever. We'll look into performers when are free. It'll take sometime since it's not a paper we are familiar with (I have only skimmed through it when it came out).

I was having a brief look at Nystromformer https://arxiv.org/pdf/2102.03902.pdf , which also seems interesting and better performance than performers. What do you think of that before performers?

vpj avatar Feb 10 '21 05:02 vpj

I can't coment on Nystromformer - but there are existing pytorch libraries for reformer models - that maybe can be cherry picked - and comented. https://github.com/search?q=performer+pytorch&type=repositories

I'm very interested in CrissCross attention / any help to bridge the linear algebra maths to the code would be amazing. GPU memory friendly High computational efficiency The state-of-the-art performance https://github.com/speedinghzl/CCNet

johndpope avatar Feb 11 '21 11:02 johndpope

https://github.com/lucidrains/nystrom-attention

johndpope avatar Feb 13 '21 06:02 johndpope

Thanks, will try to do performers when we get some free time

vpj avatar Feb 13 '21 08:02 vpj