Owen Arliawan

QLoRa Fine Tuning Mixtral 8x7b or 34b models OOM on 2x24GB Titan RTX

6

### Please check that this issue hasn't been reported before. - [X] I searched previous [Bug Reports](https://github.com/OpenAccess-AI-Collective/axolotl/labels/bug) didn't find any similar reports. ### Expected Behavior I would expect that training...

Getting deepspeed error on training completion and failing to save. if self.deepspeed_config["zero_optimization"]["stage"] == 3: AttributeError: 'Accelerator' object has no attribute 'deepspeed_config'

8

### Please check that this issue hasn't been reported before. - [X] I searched previous [Bug Reports](https://github.com/OpenAccess-AI-Collective/axolotl/labels/bug) didn't find any similar reports. ### Expected Behavior Running on Windows 10 WSL2...

[Bug]: Flash attention cannot be used on v0.5.3

7

### Your current environment ``` ./runtime.sh python env.py Collecting environment information... PyTorch version: 2.3.0 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A...

[Feature]: Is there a reason CUDA 6.1 is the minimum? Would CUDA 6.0 on the P100 not work?

8

### 🚀 The feature, motivation and pitch In the setup.py it checks for CUDA 6.1 as a minimum and that requirement is also stated in the readme. Is there a...

Is there an example on how to quantize with multiple GPUs? Is it possible to quantize Llama 3 70B with 2x3090 24GB?

1

Is there an example on how to quantize with multiple GPUs? Is it possible to quantize Llama 3 70B with 2x3090 24GB? Trying to quantize with the example quantize code...

Is there a way to run LORA extraction using multi GPU? 70B LORA extraction OOM on 24GB 3090Ti

1

Just as the title say, I am trying to extract LORA of a Llama 3.1 70B model and it OOM on a single 24GB GPU. Is there a way to...

[Bug]: LORA not working after commit e3f2ea4 "make punica kernels work with rocm" on rc_054 branch

2

### Your current environment ```python env.py Collecting environment information... PyTorch version: 2.3.0 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu...

[Bug]: Llama 3.1 outputs gibberish when --kv-cache-dtype fp8 but AWQ model works fine

4

### Your current environment Collecting environment information... PyTorch version: N/A Is debug build: N/A CUDA used to build PyTorch: N/A ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.4 LTS...

[Usage]: So can AMD NAVI GPUs be used with aphrodite? Which GPUs?

1

### Your current environment ```text The output of `python env.py` ``` ### How would you like to use Aphrodite? I want to get some AMD Navi based GPUs, but I...

[Bug]: Cannot load Mixtral GGUF model?

13

### Your current environment ``` Collecting environment information... PyTorch version: 2.3.0 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.4...