Phil Wang

[Feature suggestion] splatting list of anonymous dimensions of any length

2

Hi Alex again! :wave: Thank you again for this wonderful work :pray: I still can't get over how beautiful the abstraction is, and how much of a multiplier it is...

Suggest your favorite papers to add!

20

will start with 1. FILIP https://arxiv.org/abs/2111.07783 2. CLOOB https://arxiv.org/abs/2110.11316 3. https://arxiv.org/abs/2110.05208

[todo] checkpointing

3

make sure checkpointing works, or see if evoformer can be made reversible

[todo] BERT loss on MSA

4

needs to be added

[todo] structure module

3

in the structure module, the FAPE loss needs to be applied on every iteration, while the rotation needs to have a stop-gradient for all but the last iteration

"fixed" sampling with old <eos> method

3

@sdtblck yet another try

Separate embeddings

1

PR-ing to make sure tests work

Single head attention, decoupled LR, autoregressive auxiliary loss, and gradient accumulation

1

clean PR