haystack-core-integrations icon indicating copy to clipboard operation
haystack-core-integrations copied to clipboard

feat: Add option to select ONNXRuntime Execution Providers for fastembed

Open Fannovel16 opened this issue 1 year ago • 3 comments

Related Issues

  • In some cases, fastembed does not use GPU despite onnxruntime-gpu and fastembed-gpu are installed correctly

Proposed Changes:

  • Add option onnx_providers to set ONNXRuntime Execution Providers manually.

How did you test it?

  • WIP

Notes for the reviewer

Checklist

Fannovel16 avatar Oct 01 '24 12:10 Fannovel16

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Oct 01 '24 12:10 CLAassistant

Just a quick thought: I don't think this would work, because this integration explicitly depends on fastembed. GPU support requires fastembed-gpu, which cannot be installed in the same environment as fastembed (FastEmbed docs).

Maybe I'm wrong, but I think the whole integration needs to be reworked to support GPU. There may be several ways to do that...

Related: https://github.com/deepset-ai/haystack-core-integrations/issues/905

anakin87 avatar Oct 03 '24 16:10 anakin87

Just a quick thought: I don't think this would work, because this integration explicitly depends on fastembed. GPU support requires fastembed-gpu, which cannot be installed in the same environment as fastembed (FastEmbed docs).

Maybe I'm wrong, but I think the whole integration needs to be reworked to support GPU. There may be several ways to do that...

Related: #905

@anakin87 On Google Colab, I installed fastembed-gpu and correct onnxruntime-gpu but it does not pick up GPU by default, even if "CUDAExecutionProvider" is available. Explicily stating providers fix that. I've been using this fork on my project rn

#@title Install library
!pip install git+https://github.com/deepset-ai/haystack "datasets>=2.6.1" "sentence-transformers>=3.0.0" markdown-it-py mdit_plain
!pip install langchain-text-splitters instructor

%cd /content
!git clone https://github.com/Fannovel16/haystack-core-integrations/
%cd /content/haystack-core-integrations/integrations/fastembed
!pip install .
%cd /content/haystack-core-integrations/integrations/qdrant
!pip install .
%cd /content

!pip install "onnxruntime-gpu<=1.17" -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
!pip install fastembed-gpu

Fannovel16 avatar Oct 04 '24 02:10 Fannovel16