Raj Kachhadiya

icon company ConcertAI icon location Bangalore icon location NLP Data scientist @ConcertAI || ex - ML Research Intern @Bosch || MTech ECE @IIITBangalore

Results 2 comments of Raj Kachhadiya

train 1B model on 32G V100 gpu ，flash_attention not support, any one train 1B model on V100? A100 cost expensive

Hello @Weiyun1025 Thanks for your reply. after this change I am still getting error ` RuntimeError: FlashAttention only supports Ampere GPUs or newer. ` For your reference I shared my...

[Bug] FlashAttention Error During InternVL Fine-tuning on Tesla T4 GPU

I saw in the code they changed the parameter with "flash_attention_2" directly they are not using the config as a parameter so this repo won't help you anymore, I didn't...