tortoise.cpp icon indicating copy to clipboard operation
tortoise.cpp copied to clipboard

Published 20 hours ago • verified publisher icon balisujohn

AMD GPUs

Open fakerybakery opened this issue 1 year ago • 5 comments

Hi, I saw this project currently only supports Cuda. I was wondering if it might be possible to use HIPIFY to make it work on AMD GPUs. Do you know if this would be possible? Thank you!

fakerybakery avatar Jan 08 '24 02:01 fakerybakery

So I'm using the GGML library, which I think may already support AMD ROCm, though for now I'm just compiling for cuda. If you have an AMD card with ROCm, I'm happy to help you get started on working on this, but I can't do it myself since I don't have an AMD card. https://github.com/ggerganov/ggml

balisujohn avatar Jan 08 '24 02:01 balisujohn

Thanks! Do you know if it works on CPU?

fakerybakery avatar Jan 08 '24 03:01 fakerybakery

I have an experimental CPU branch, but the numbers don't match for now because I changed the behavior of some cuda ops without making the equivalent change for the CPU version of the OP. it will take a lot of work to fix, and probably be so slow it's far less useful, so it's low priority for me.

https://github.com/balisujohn/tortoise.cpp/tree/exp-processor

balisujohn avatar Jan 08 '24 03:01 balisujohn

I should add that I'm supportive of adding support for all platforms including CPU, and will accept contributions towards support for any platform.

balisujohn avatar Jan 08 '24 06:01 balisujohn

@fakerybakery CPU support is now added.

balisujohn avatar Jun 25 '24 07:06 balisujohn