metaformer

metaformer copied to clipboard

Caformer finetuning. Adapter/LoRa?

5

Hey, great job with this repo. Caformer with 100M parameters is really powerful, though I am struggling with the finetuning due to hardware limitations. Did you already make experiments with...

kacwin

metaformer has no positional encoding?

1

I notice that Metaformer has no positional encoding(PE) either in the attention layers or at the model input, does this affect the performance? Is the positional encoding not necessary? What...

bio-mlhui

Question regarding Equation 5 Random Mixing in the Paper

1

Should the matrix multiplication not be swapped? $$ RandomMixing(X) = XW_R \Rightarrow RandomMixing(X) = W_RX $$ I think the dimension don`t work for the original equation because you are multiplying...

f-fuchs