Ashwin J S comments

Repositories
Issues
Comments

Results 3 comments of


                                            Ashwin J S

ImportError: Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes: `pip install -i https://pypi.org/simple/ bitsandbytes`

> fwiw, downgrading to a lower version of transformers helped resolve the issue for me (4.38.2 to 4.31.0). This worked, Thanks !!

NVIDIA AMMO documentation

Hi @dmitrymailk , I am also exploring ways to run 4bit quantized encoder - decoder model in tensorrt-llm. Where you able to make any progress on that front ?

Fail to build Llama-3-70B-Instruct with w4a16

I am getting this error now when I am trying to convert a fine tuned llama3 8b gptq safetensor. Does the patch f430a4b addresses gptq ?