jianyuheng comments

Repositories
Issues
Comments

Results 4 comments of


                                            jianyuheng

RuntimeError: self and mat2 must have the same dtype

@yuhuixu1993 same problem here. > @M-Elfeki我检查了我的依赖项，我使用了 auto-gotq==0.3.0.dev0，你可以尝试 0.3.0

RuntimeError: self and mat2 must have the same dtype

The error no longer appears when I modify these two lines of code. @yuhuixu1993 https://github.com/yuhuixu1993/qa-lora/blob/8791e08929fee2bf015a9bbc0ebaaefd0c9cf2a5/qalora.py#L300 `model.config.torch_dtype=torch.float16` https://github.com/yuhuixu1993/qa-lora/blob/8791e08929fee2bf015a9bbc0ebaaefd0c9cf2a5/qalora.py#L340 `module = module.to(torch.float16) `

Adapter dimensions question

This bug is still not solved. @yuhuixu1993

Question about inference speed

I know that exllama has some optimizations on cuda core. In fact, I mainly want to know that if awq uses GPU optimization technology, will the performance of awq be...