fastembed-go icon indicating copy to clipboard operation
fastembed-go copied to clipboard

Published 20 hours ago • verified publisher icon Anush008

Using GPU

Open msj121 opened this issue 11 months ago • 4 comments

Looking through the code I didn't see a setting for using Cuda/GPU, is this done automatically, or is there a setting? Tried looking in the Onnx project, but got a little confused.

Thanks.

msj121 avatar Mar 14 '24 22:03 msj121

Hi. We could add a configuration option for GPU, like FastEmbed for the other languages. But since I don't have access to a GPU instance, I can't verify it. If you do, please consider contributing.

Anush008 avatar Mar 15 '24 03:03 Anush008

Ok - I do have indirect access to a computer with a gpu. Let me try and look around the code etc... If you have any advice lmk.

msj121 avatar Mar 15 '24 17:03 msj121

There's an execution providers option. https://github.com/Anush008/fastembed-go/blob/bbcc4da928d23eea76be309148ad1d298b9c23af/fastembed.go#L61 that is not being used.

You'd need to verify if we can pass and make it work with CUDA for example. We're using https://github.com/yalue/onnxruntime_go to interface with ONNX.

Anush008 avatar Mar 15 '24 18:03 Anush008

Probably would have to change the type of the option from string[] to the one used by onnxruntime_go for execution providers.

Anush008 avatar Mar 15 '24 18:03 Anush008

Closing since stale. Please feel free to re-open.

Anush008 avatar Aug 24 '24 08:08 Anush008