[PROPOSAL] Add support for asymmetric embedding models to neural-search

[br3no]

What/Why

What are you proposing?

With https://github.com/opensearch-project/ml-commons/issues/1799 OpenSearch can now use asymmetric embedding models such as e5. Asymmetric embedding models differ from symmetric ones by embedding queries and passages differently. Many of them require the text to be prefixed by special strings (e.g. query: and passage: for the e5 model family). The aforementioned issue documents the work that led to ml-commons supporting serving these kinds of model. There, new model configurations have been added, a special new AsymmetricTextEmbeddingParameters type was introduced and the correct handling of inference for these models has been implemented. Here I want to propose to add support for this in the neural-search plugin.

What users have asked for this feature?

I'm a user and would like to have this feature. 😉 Many of the best performing embedding models are asymmetric ones (cf. MTEB board https://huggingface.co/spaces/mteb/leaderboard). For OpenSearch it is important to keep up with the latest developments in information retrieval research and give users the freedom to choose as many different models as possible.

What problems are you trying to solve?

Neural-search should be able to use the newly introduced capability in ml-commons.

What is the developer experience going to be?

My proposal will not affect any API. The idea is to have neural-search automatically detect that a particular model being used is asymmetric (it is possible to infer this information from the model configuration) and then call the inference method with the right parameters. In the embedding processor, we need to add the information that we are embedding a passage; in the query builder, that we are embedding a query. That's it.

Are there any security considerations?

No.

Are there any breaking changes to the API

No.

What is the user experience going to be?

The user experience will not change in any way.

Are there breaking changes to the User Experience?

No.

Why should it be built? Any reason not to?

Not building this reduces the attractiveness of OS for dense retrieval. Other vector DBs already support these models.

What will it take to execute?

The ground work has been done in ml-commons, here we only need to build on top of it. I will open a PR soon on which we can base the discussion.

Any remaining open questions?

I can't think of any.