russellsparadox

Results 17 comments of russellsparadox

multi-gpu bloomz-7b1 fails: RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:7

I tried to follow [this suggestion](https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt2-sentiment_peft.py#L143-L157) but got the following error:

multi-gpu bloomz-7b1 fails: RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:7

https://github.com/huggingface/peft/issues/269#issuecomment-1498868232

multi-gpu bloomz-7b1 fails: RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:7

I uninstall and reinstalled everything with: ``` pip install git+https://github.com/huggingface/peft.git pip install git+https://github.com/huggingface/accelerate.git pip install git+https://github.com/huggingface/transformers.git ``` It didn't help and it gives familiar: `RuntimeError: expected scalar type Half but...

RuntimeError: The size of tensor a (2048) must match the size of tensor b (2131) at non-singleton dimension 3

Hi @Jblauvs , Do you mind sharing here particular lines of code you changed?