Support for AMD GPUs in the Llama Recipes notebook quickstart

[PatchouliPatch]

🚀 The feature, motivation and pitch

So I'm using an RX 7900 XTX with PyTorch 2.2.0.dev20231005+rocm5.7. From the error message below, it seems that it's a problem with bitsandbytes not supporting AMD GPUs (yet). I would like to ask for a version of the Quick Start guide that would support AMD GPUs as well, not just Nvidia GPUs.

I get the following error when I try to run model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", load_in_8bit=True, device_map='auto',torch_dtype=torch.float16):

Traceback (most recent call last): File "/home/gabriel/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1282, in _get_module return importlib.import_module("." + module_name, self.name) File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/home/gabriel/.local/lib/python3.10/site-packages/transformers/integrations/bitsandbytes.py", line 11, in import bitsandbytes as bnb File "/home/gabriel/.local/lib/python3.10/site-packages/bitsandbytes/init.py", line 6, in from . import cuda_setup, utils, research File "/home/gabriel/.local/lib/python3.10/site-packages/bitsandbytes/research/init.py", line 1, in from . import nn File "/home/gabriel/.local/lib/python3.10/site-packages/bitsandbytes/research/nn/init.py", line 1, in from .modules import LinearFP8Mixed, LinearFP8Global File "/home/gabriel/.local/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in from bitsandbytes.optim import GlobalOptimManager File "/home/gabriel/.local/lib/python3.10/site-packages/bitsandbytes/optim/init.py", line 6, in from bitsandbytes.cextension import COMPILED_WITH_CUDA File "/home/gabriel/.local/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 13, in setup.run_cuda_setup() File "/home/gabriel/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 120, in run_cuda_setup binary_name, cudart_path, cc, cuda_version_string = evaluate_cuda_setup() File "/home/gabriel/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 341, in evaluate_cuda_setup cuda_version_string = get_cuda_version() File "/home/gabriel/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 311, in get_cuda_version major, minor = map(int, torch.version.cuda.split(".")) AttributeError: 'NoneType' object has no attribute 'split'

Alternatives

No response

Additional context

No response

[HamidShojanazeri]

@PatchouliPatch we are looking into AMD GPUs/ will keep you posted, however the bits&bytes issue with AMD you might help to open an issue on their repo.

[raghukiran1224]

We have tried this script on AMD GPUs and it works for LoRa and full fine tuning. We have not tried bits-n-bytes.

[jinsong-mao]

We have tried this script on AMD GPUs and it works for LoRa and full fine tuning. We have not tried bits-n-bytes.

can u share us your flow/script on how to fix the bit&bytes issue?

thanks