Support mac M1 GPU (mps)

[slavag]

Hi, I saw that lama-cpp-python is supporting Mac M1 GPU since version 0.1.64 and lama.cpp as well already supporting metal. Is there any plans to upgrade to use those versions to support as well Mac metal ?

Thanks

[ChristianWeyer]

Heavily needed! :-)

[m-roberts]

Would love to see this

[pseudotensor]

@Mathanraj-Sharma offered to take a look. He has a MAC with M1 and 16GB ram.

Moved M1 mac specific issue out of https://github.com/h2oai/h2ogpt/issues/159 and into this issue.

[pseudotensor]

@Mathanraj-Sharma See: https://github.com/PromtEngineer/localGPT#m1m2-macbook-users

M1/M2 Macbook users:

1- Follow this page to build up PyTorch with Metal Performance Shaders (MPS) support. PyTorch uses the new MPS backend for GPU training acceleration. It is good practice to verify mps support using a simple Python script as mentioned in the provided link.

2- By following the page, here is an example of you may initiate in your terminal

xcode-select --install
conda install pytorch torchvision torchaudio -c pytorch-nightly
pip install chardet
pip install cchardet
pip uninstall charset_normalizer
pip install charset_normalizer
pip install pdfminer.six
pip install xformers

3- Create a new verifymps.py in the same directory (localGPT) where you have all files and environment.

import torch
if torch.backends.mps.is_available():
    mps_device = torch.device("mps")
    x = torch.ones(1, device=mps_device)
    print (x)
else:
    print ("MPS device not found.")

4- Find instructor.py and open it in VS Code to edit.

The instructor.py is probably embeded similar to this:

file_path = "/System/Volumes/Data/Users/USERNAME/anaconda3/envs/LocalGPT/lib/python3.10/site-packages/InstructorEmbedding/instructor.py"

You can open the instructor.py and then edit it using this code:

Open the file in VSCode

subprocess.run(["open", "-a", "Visual Studio Code", file_path])

Once you open instructor.py with VS Code, replace the code snippet that has device_type with the following codes:

     if device is None:
        device = self._target_device

    # Replace the line: self.to(device)

    if device in ['cpu', 'CPU']:
        device = torch.device('cpu')

    elif device in ['mps', 'MPS']:
        device = torch.device('mps')

    else:
        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    self.to(device)

[slavag]

@pseudotensor @Mathanraj-Sharma I saw in the recent change in h2ogpt that MPS is supported, but I'm unable to llama using MPS. This is screenshot on loading (lates code)

Thanks

[pseudotensor]

@Mathanraj-Sharma , any ideas?

[pjaol]

Can you update the MAC OS readme to indicate mps support isn't available https://github.com/h2oai/h2ogpt/blob/main/docs/README_MACOS.md

Spent a while trying to figure out why it wasn't using any GPU

[pseudotensor]

@pjaol Hi, MPS support was added. This issue should be closed. @Mathanraj-Sharma added it.

[pseudotensor]

https://github.com/h2oai/h2ogpt/blob/main/docs/README_MACOS.md#gpu-mps-mac-m1

[slavag]

@pseudotensor But, it's actually not using MPS, you can see it in the screenshot I sent. Thanks

[pseudotensor]

Yes, I've poked @Mathanraj-Sharma . Let's make new issue for that. He got it working, so something off. He can check if really using GPUs etc.

[pseudotensor]

@slavag FYI I think @Mathanraj-Sharma tested non-llama models like h2ogpt models.

However, to get GPU support for llama, one needs to specifically decide that and compile llama cpp python properly, like mentioned here:

https://github.com/h2oai/h2ogpt/blob/main/docs/README_GPU.md#gpu-with-llama

@Mathanraj-Sharma Please check if this works for MAC M1 and add/copy that to the MAC readme.

[Mathanraj-Sharma]

Found out the missing piece will update the readme in a new PR