gemma.cpp icon indicating copy to clipboard operation
gemma.cpp copied to clipboard
verified publisher icon google

GPU Support

Open jerrymatjila opened this issue 1 year ago • 2 comments

Is there a way to use GPU or is this on the roadmap.

jerrymatjila avatar Feb 23 '24 10:02 jerrymatjila

GPUs are not supported yet, though we have done a bit of prototyping. Speaking personally, I believe GPUs are well-supported by other inference frameworks and less suitable here given our focus on "experimentation and research".

jan-wassenberg avatar Feb 23 '24 13:02 jan-wassenberg

I'm looking into options with metal shaders/vulkan/webgpu, if anyone wants to collaborate please reach out.

austinvhuang avatar Feb 23 '24 13:02 austinvhuang

Closing for now but will update here when there's more info on this.

austinvhuang avatar Feb 24 '24 05:02 austinvhuang