AttributeError: 'ExLlamaV2Attention' object has no attribute 'device_idx'

[dsidlo]

Expected Behavior

• After entering prompt.
• LLM Response is returned.

Current Behavior

• After Entering Prompt...

warmup for generating up to 3951 tokens Couldn't generate Traceback (most recent call last): File "/home/dsidlo/workspace/lollms-webui/zoos/bindings_zoo/bs_exllamav2/init.py", line 371, in generate self.generator.warmup() File "/home/dsidlo/anaconda3/envs/lollms/lib/python3.11/site-packages/exllamav2/generator/base.py", line 37, in warmup self.model.forward(input_ids, cache = None, input_mask = None, preprocess_only = True) File "/home/dsidlo/anaconda3/envs/lollms/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/dsidlo/anaconda3/envs/lollms/lib/python3.11/site-packages/exllamav2/model.py", line 512, in forward result, last_state = self._forward(input_ids = input_ids, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/dsidlo/anaconda3/envs/lollms/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/dsidlo/anaconda3/envs/lollms/lib/python3.11/site-packages/exllamav2/model.py", line 611, in _forward device = _torch_device(module.device_idx) ^^^^^^^^^^^^^^^^^ AttributeError: 'ExLlamaV2Attention' object has no attribute 'device_idx'

Steps to Reproduce

Please provide detailed steps to reproduce the issue.

1. Installed model: Qwen1.5-4B-Chat-GPTQ-Int8
2. Enter prompt: Hello?

Possible Solution

Do I have the wrong version of exllamav2? My version: 0.0.13+cu121

Context

Using Python 3.11.17 Cuda Version: 12.0

Screenshots

[ParisNeo]

I spent the day trying to fix thix. Reinstall lollms then reinstall exllamav2 binding. It should work now.