vit.cpp icon indicating copy to clipboard operation
vit.cpp copied to clipboard
verified publisher icon staghado

is it possible to use cuda

Open twmht opened this issue 1 year ago • 4 comments

Hi,

is it possible to use cuda for inference?

twmht avatar Jan 11 '24 10:01 twmht

It is possible : refer to the GGML library.

staghado avatar Jan 11 '24 11:01 staghado

yeah i know ggml supports cuda, what i mean is do you have any plan to add options(https://github.com/staghado/vit.cpp?tab=readme-ov-file#run) for cuda?

twmht avatar Jan 11 '24 12:01 twmht

I can add it if that's something you need. One issue is that i do not have a local GPU so it will be hard to test. would you like to test? you can also open a PR if you like!

staghado avatar Jan 11 '24 12:01 staghado

@staghado

unfortunately i am not familir with ggml now, but i can test if you can help to submit a PR:)

twmht avatar Jan 11 '24 12:01 twmht