Kyle Herndon comments

Repositories
Issues
Comments

Results 3 comments of


                                            Kyle Herndon

HIP runtime memory issue for Llama 3.1 70B F16.

The device I'm using has approximately 200GB of memory. I updated the filebin with two additional files. I halved the number of attention layers in the model so the model...

HIP runtime memory issue for Llama 3.1 70B F16.

Added two more files to the [filebin](https://filebin.net/u4gmgdsh5s6ks6lr) with just one attention layer and it did finally run. I would think this would put an upper bound on the remaining additional...

HIP runtime memory issue for Llama 3.1 70B F16.

Same general error when running with those flags, at least on 405b. @aviator19941 said he would try out 70b.