Yu-Chien Lin

Results 3 comments of Yu-Chien Lin

Modifying ALiBi for Encoder-Attention or Cross-Attention

Hi @ofirpress , I'm trying to use ALiBi for machine translation. It works well if I only apply ALiBi in the decoder (BLEU is about 55). But the performance gets...

Modifying ALiBi for Encoder-Attention or Cross-Attention

> Hi @lyc2001: Try applying ALiBi in the encoder with variation 1 or 2. Use decoder alibi as normal. Don't apply any kind of bias or position embedding to the...

Modifying ALiBi for Encoder-Attention or Cross-Attention

Hi @ofirpress , Thank you for your reply! I'll try to apply it on shorter length sentences and see what happens.