device set bug

[Yuang-Deng]

System Info

optimum                  1.16.1

Who can help?

@philschmid

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [X] My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

model_id = "facebook/opt-125m" tokenizer = AutoTokenizer.from_pretrained(model_id) quantization_config = GPTQConfig(bits=4, dataset=["c4", "c4", "c4"], tokenizer=tokenizer)

model = AutoModelForCausalLM.from_pretrained(model_id, device_map="cuda:5", quantization_config=quantization_config)

print()

Expected behavior

optimum/gptq/quantizer.py line 429 data[k] = v.to(0)

Why is it fixed at 0? When setting device_map for the model, an error occurs that the input and model are not on the same device. Is this a bug?

[fxmarty]

Hi @Yuang-Deng happy to have a look, could you provide a full reproduction? I think your code snippet is truncated.