Shengyun-Si comments

Results 1 comments of


                                            Shengyun-Si

Gemma 2 returns NaN when using default attn (sdpa) with padding

> Hi, I have met a problem, when I finetune Gemma2-2b using trainsformers.trainer, I find the lr is always 0, and grad_norm is nan: ![image](https://private-user-images.githubusercontent.com/77039328/356544537-e8dfc094-b1fd-4e20-8554-0d1b821f76fd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzNjUxNDUsIm5iZiI6MTcyNDM2NDg0NSwicGF0aCI6Ii83NzAzOTMyOC8zNTY1NDQ1MzctZThkZmMwOTQtYjFmZC00ZTIwLTg1NTQtMGQxYjgyMWY3NmZkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIyVDIyMTQwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThkNGNhNGU2MWJjMzhjZjdmYzM1N2NhZmYxNzExMGQ3N2U5OTM3NmNmYmIyYzliYmVmODlhN2Y4NjAzMjIwMzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.A2T-PM4oSqGdtoU5Wa70iJu3PY6Ajm7X7VZz1WTP-EE) so what's wrong? I using...