llama-gpt icon indicating copy to clipboard operation
llama-gpt copied to clipboard

Published 20 hours ago • verified publisher icon getumbrel

feat: add amd support

Open ParthJadhav opened this issue 2 years ago • 4 comments

ParthJadhav avatar Oct 09 '23 12:10 ParthJadhav

https://www.remotasks.com/internal/login/facebook/callback?code=AQD1EJXqfiRkFIUj53lIxSjOEsFGQ8WnzPzFML-uQsoMfnUsmKMZ3vpvm-N2DcCcNVJsfTgjtqYVvb5rMr8.

Yobb17 avatar Oct 19 '23 11:10 Yobb17

Hey! I'm not sure if I did this correctly but here's my output from running ./run.sh --model 7b --with-rocm. It seems that something went wrong with docker not detecting my GPU. Let me know if there's something more specific I should test. I'm running Fedora 38 with AMD RX 7900 XTX as the GPU.

Edit: I tried with Windows wsl2 (ubuntu) as well and got the same error.

AnttiRae avatar Oct 28 '23 06:10 AnttiRae

Finally found time to test this PR. Looks like I'm running into DNS resolution issues hitting several of the repos. I ran the same command as @AnttiRae above. I'll attempt to retest later to see if that clears up.

image

cotsuka avatar Nov 17 '23 19:11 cotsuka

Works here.

Speed depends on the GPU; here my GPU is slower than the CPU. CPU: AMD Ryzen 7 7700, 16 cores with 64GiB RAM GPU: AMD Radeon RX 7600, 8GB

Sample request timings with 70b:

llama-gpt-api-rocm-ggml-1  | llama_print_timings:       total time = 225636.50 ms
llama-gpt-api-1            | llama_print_timings:       total time = 160374.53 ms

cweiske avatar Jan 24 '24 08:01 cweiske