massive-activations

massive-activations copied to clipboard

Training only on 2B tokens (openwebtext)

3

Hi ! Interesting work on the role of explicit bias! I was wondering what training settings got you an eval PPL ~3.04. The paper mentions that 50K iterations are required...

Nandan91

what if set all layers' massive activations to their mean value?

1

interesting work! i have a question as it in the title, do you conducte an experiment like that? what's the result? thanks.

yyfcc17

How to get the mean value of massive activation

1

1. How to get the mean value of massive activation？e.g. 2546.8/-1502.0 in hook.py 2. Mean value is still large, what is the difference between using the mean value and using...

pengyao96

an -> a

eltociear

Which layer's activation is used?

1

Hello, This is great work! And I wonder about the layer that the analyzed activations are from. The last layer?

iyupan

the standard deviation of the activation

3

Hello, I am interested in the standard deviation of the activation and would like to know how the variance is calculated. Here are a few methods: 1. Calculate the variance...

Cooperx521