larin92 comments

Repositories
Issues
Comments

Results 3 comments of


                                            larin92

2 train steps for a single batch?

Yes, but this can be accomplished in one execution of session.run. It looks like one of those lines should be commented out at a time

Feature Request: Quantized Mixtral

yes please, support for pre-quantized models from HuggingFace would be great. i'm not even sure i can use multi-gpu setup for DIY quantization using TensorRT-LLM, as this file doesn't have...

Mixtral convert error: t() expects a tensor with <= 2 dimensions, but self is 3D

> I managed to quantize Mixtral 8x7B to 4 bpw. > > I first tried running this command: > > ```shell > model="models--mistralai--Mixtral-8x7B-Instruct-v0.1" > model_dir="/models/$model" > model_chkpt_dir="/models/$model--trt-chkpt" > > python3...