gemma icon indicating copy to clipboard operation
gemma copied to clipboard
verified publisher icon google-deepmind

Sliding Window Attention

Open aniquetahir opened this issue 1 year ago • 3 comments

I noticed this version does not contain sliding window attention needed for Gemma 2 models.

aniquetahir avatar Jul 09 '24 21:07 aniquetahir

We are working to release the Gemma 2 related code soon, sorry for the delay

gustheman avatar Jul 16 '24 13:07 gustheman

any update?

Mddct avatar Jul 24 '24 04:07 Mddct

Hey folks, Thank you for your interest

It will be released soon and contain all the v2 updates such as sliding window attention and GQA. I'll update this issue in the next two weeks!

canyon289 avatar Jul 24 '24 14:07 canyon289

Hi @aniquetahir,

Sliding window attention has been added to the Gemma 2 models. Could you please check it.

Thank you.

Gopi-Uppari avatar Oct 16 '24 05:10 Gopi-Uppari

Hi @aniquetahir,

Sliding window attention has been added to the Gemma 2 models. Could you please check it.

Thank you.

This is great!

aniquetahir avatar Oct 16 '24 15:10 aniquetahir