EAGLE icon indicating copy to clipboard operation
EAGLE copied to clipboard
verified publisher icon SafeAILab

Why does EAGLE remove the input_layernorm of llama?

Open bisunny opened this issue 1 year ago • 2 comments

image

bisunny avatar May 26 '24 09:05 bisunny

The base model has a layer normalization (layernorm) layer before the LM head. Since the feature sequence has already been normalized, we do not use layer normalization.

Liyuhui-12 avatar Jun 09 '24 08:06 Liyuhui-12

The base model has a layer normalization (layernorm) layer before the LM head. Since the feature sequence has already been normalized, we do not use layer normalization.

It is true that the base model has a layer normalization (layernorm) layer before the LM head, but this has nothing to do with EAGLE remove the input_layernorm of llama. I guess this is a trick to help improve Eagle accuracy?

haiduo avatar Jul 16 '24 09:07 haiduo