michaelfeil
Allow to pass SentenceTransformers `task` and `prompt`
Feature request
I want to use https://huggingface.co/jinaai/jina-embeddings-v3 with via infinity server and client, but I cant find an example to run it with task as in SentenceTransformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("jinaai/jina-embeddings-v3", trust_remote_code=True)
task = "retrieval.query"
embeddings = model.encode(
["What is the weather like in Berlin today?"],
task=task,
prompt_name=task,
)
Motivation
It will not be practical to use infinity for jinaai embeddings (and some others) without this functionality. Actually, we can prefix our input text with relevant prompt, but we should not.
Your contribution
I may be the first who will test and give feedback. If you hint me a contribution start point, I can try to cook a PR.
Hi,
The easiest way would be to provide the task and prompt at the startup of the model. Would this cover your use case?
Prompt could be passed in and prepended here.
For the task, you would pass it here.
This PR does similar changes for the dimensions param.
Hi,
The easiest way would be to provide the
taskandpromptat the startup of the model. Would this cover your use case?Prompt could be passed in and prepended here.
For the task, you would pass it here.
This PR does similar changes for the dimensions param.
I guess task and prompt should be passed to embedding, not in the model init stage: same model can be used for two tasks: retrieval.query and retrieval.passage.
In this case you would want to add the task and prompt as additonal parameters to the request. You could modify _OpenAIEmbeddingInput to take task and prompt. Then pass it through to the methods as linked in my previous comment.
One challenge I can see is we need to separate the batches based on the task to pass them through the sentence transformer.
The LoRA adapters are the core of jina-embeddings-v3, without this feature, one cannot claim that the model is supported.
@dantetemplar did you solve it?
Just prefix your text with correct prefix from your model's config on huggingface
@dantetemplar this model requires task specification, not simple prefix unfortunately
So, does your model change inference path (what head to use) depending on that parameter?
If yes, dunno what to do actually except patching this framework
@dantetemplar looking at the code of models more closely, you are right it's just a prefix. So one can simply provide it.