haystack-core-integrations
haystack-core-integrations copied to clipboard
feat: Add option to select ONNXRuntime Execution Providers for fastembed
Related Issues
- In some cases, fastembed does not use GPU despite
onnxruntime-gpuandfastembed-gpuare installed correctly
Proposed Changes:
- Add option
onnx_providersto set ONNXRuntime Execution Providers manually.
How did you test it?
- WIP
Notes for the reviewer
Checklist
- I have read the contributors guidelines and the code of conduct
- I have updated the related issue with new insights and changes
Just a quick thought: I don't think this would work, because this integration explicitly depends on fastembed.
GPU support requires fastembed-gpu, which cannot be installed in the same environment as fastembed (FastEmbed docs).
Maybe I'm wrong, but I think the whole integration needs to be reworked to support GPU. There may be several ways to do that...
Related: https://github.com/deepset-ai/haystack-core-integrations/issues/905
Just a quick thought: I don't think this would work, because this integration explicitly depends on
fastembed. GPU support requiresfastembed-gpu, which cannot be installed in the same environment asfastembed(FastEmbed docs).Maybe I'm wrong, but I think the whole integration needs to be reworked to support GPU. There may be several ways to do that...
Related: #905
@anakin87 On Google Colab, I installed fastembed-gpu and correct onnxruntime-gpu but it does not pick up GPU by default, even if "CUDAExecutionProvider" is available. Explicily stating providers fix that. I've been using this fork on my project rn
#@title Install library
!pip install git+https://github.com/deepset-ai/haystack "datasets>=2.6.1" "sentence-transformers>=3.0.0" markdown-it-py mdit_plain
!pip install langchain-text-splitters instructor
%cd /content
!git clone https://github.com/Fannovel16/haystack-core-integrations/
%cd /content/haystack-core-integrations/integrations/fastembed
!pip install .
%cd /content/haystack-core-integrations/integrations/qdrant
!pip install .
%cd /content
!pip install "onnxruntime-gpu<=1.17" -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
!pip install fastembed-gpu