orellavie1212

Issues with Upgraded Pip Libs for LoRA Weights in 4-bit and 8-bit Training

9

Dear Support Team, I recently upgraded my pip libraries, including transformers, peft, accelerate, and bitsandbytes, to support 4-bit training as opposed to the original 8-bit training. After doing so, I...

[Feature]: bitsandbytes support

17

### 🚀 The feature, motivation and pitch Bitsandbytes 4bit quantization support. I know many want that, and also it is discuused before and marked as unplaned, but after I looked...

problem of fuzz.ratio with newer ver (22) of python-Lev. distance

2

Hey, After testing failure of fuzz.ratio at the detection of basic sim. match, I found that the newer ver of python-Levenshtein(0.22.0) makes the fuzz.ratio output wrong answer when installing it...

Bug - mixtral qlora(after b&b peft train) quantization broadcast problem

3

Hello, Wanted to quantize the model via awq after a merged qlora b&b nf4 mixtral moe. the error is: ``` self._search_best_scale(self.modules[i], **layer) File "/home/access/anaconda3/envs/sec_qlora_replicate/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args,...

[Bug]: tensor parallel (of 4 cards) gives bad answers in version 0.5.1 and later (compared to 0.4.1) with gptq marlin kernels (compared to gptq)

6

### Your current environment sagemaker ml.g5.12xlarge instance (4 instances of a10g 24gb) container is 763104351884.dkr.ecr.us-west-2.amazonaws.com/djl-inference:0.27.0-deepspeed0.12.6-cu121 from https://github.com/aws/deep-learning-containers/blob/master/available_images.md ### 🐛 Describe the bug from vllm import LLM, SamplingParams question = "what...